Patent Description:
<CIT> relates to partially replicated distributed database with multiple levels of remote clients. A method of maintaining a partially replicated database is provided, where updates are made to a central database, or to another partially replicated database, that are selectively propagated to the partially replicated database. Updates are propagated to a partially replicated database if the owner of the partially replicated database is deemed to have visibility to the data being updated. Visibility is determined by use of predetermined rules stored in a rules database. Differences of merge processing versus update processing are first, that the input to a merge processor is not an update entered directly by a user, but rather is a log file that is obtained from a computer remote from the computer where the merge is executing. Second, merge processing does not produce a log when performed at a node. Third, merge processing must be capable of detecting and resolving multiple conflicting transactions. A collision is detected and corrected by comparing the value in the database to the value that the merge log reflects as being the previous value in the database node. If the two values do not match, the merge processor rejects the transaction and generates a corrective transaction to be sent to the node from which the conflicting transaction originated.

<NPL>, states that when a database has to make an update, internally (at low level) it makes a delete and then an insert with the updated field. This statement is described as not true.

It is the object of the present invention to improve performance of a global index accessed by several processes.

Some technologies described herein are directed to the technical activity of automatically detecting unauthorized attempts to access items stored in a computing system, thereby reducing the risk of undetected intrusions. Some teachings are directed to protecting storage accounts from cyberattacks by detecting unrecommended access through an adaptation of collaborative filtering for use in cybersecurity. Technical mechanisms are described for assessing intrusion risk based not necessarily on what is considered baseline or normal behavior for a particular system or a particular user, but instead based at least in part on a measure of what accesses are to be expected from multiple accessors to a given storage item. Specific technical tools and techniques are described here in response to the challenge of providing defense-in-depth by way of layered intrusion detection or layered intrusion prevention. Other technical activities pertinent to teachings herein will also become apparent to those of skill in the art.

Some embodiments described herein adapt collaborative filtering from its use in recommendation systems to usage instead for cyberattack intrusion detection. As to the technical adaptations made, rather than operational data and structures representing people and products they might buy, some embodiments operate with accessor IDs and storage.

Software processes access data stored in a storage medium, such as files in a file system. To access this data quickly and efficiently, a global index may be created that maps unique keys to stored data. For example, in systems where there is a portioning of work across multiple concurrently running processes, indices are often created to find partitioned data sets. For example, in a system where processes manage siloed data, a global index may be created to maintain what data is available on the system for proper routing and consistent management of data.

One challenge in managing such an index is how to maintain the global index with a plurality of processes accessing the index with minimal latency and locking impact while maintaining atomicity, consistency, isolation, and durability (ACID) guarantees of the global index. Accordingly, embodiments described herein address this and other technical problems using a multi-process journaling index.

One or more embodiments are described and illustrated in the following description and accompanying drawings. These embodiments are not limited to the specific details provided herein and may be modified in various ways. Furthermore, other embodiments may exist that are not described herein. Also, the functionality described herein as being performed by one component may be performed by multiple components in a distributed manner. Likewise, functionality performed by multiple components may be consolidated and performed by a single component. Similarly, a component described as performing particular functionality may also perform additional functionality not described herein. For example, a device or structure that is "configured" in a certain way is configured in at least that way, but may also be configured in ways that are not listed. Furthermore, some embodiments described herein may include one or more electronic processors configured to perform the described functionality by executing instructions stored in non-transitory, computer-readable medium. Similarly, embodiments described herein may be implemented as non-transitory, computer-readable medium storing instructions executable by one or more electronic processors to perform the described functionality. As used in the present application, "non-transitory computer-readable medium" comprises all computer-readable media but does not consist of a transitory, propagating signal. Accordingly, non-transitory computer-readable medium may include, for example, a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, a ROM (Read Only Memory), a RAM (Random Access Memory), register memory, a processor cache, or any combination thereof.

In addition, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. For example, the use of "including," "containing," "comprising," "having," and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. The terms "connected" and "coupled" are used broadly and encompass both direct and indirect connecting and coupling. Further, "connected" and "coupled" are not restricted to physical or mechanical connections or couplings and can include electrical connections or couplings, whether direct or indirect. In addition, electronic communications and notifications may be performed using wired connections, wireless connections, or a combination thereof and may be transmitted directly or through one or more intermediary devices over various types of networks, communication channels, and connections. Moreover, relational terms such as first and second, top and bottom, and the like may be used herein solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.

As described above, indices may be generated to manage access to stored data among a plurality of processes. For example, a client device may access data via a cloud service and a local copy of the accessed data may be stored on the client device. Multiple processes executed by the client device may need access to the local copies and may each access the data through a global index. Although the global index provides fast access to the locally-stored data, access and manipulation of the global index must be managed to prevent conflicts and delays.

Generally, these problems are solved by locking all access to the global index across all processes or re-trying transactions when conflicts occur. Both of these solutions are problematic performance-wise due to increased input/output usage and processing usage. These solutions can also lead to bugs due to unexpected re-trying of transactions and the business logic associated with such re-trying.

Accordingly, the systems and methods described herein provide a multi-process journaling index wherein each process maintains a journal of activities performed relative to a global index. Periodically, the journal is applied to global index. Thus, the multi-process journaling index resolves conflicts across access to the global index and supports not-in-place indices, which is solid state storage (drive or disk) (SSD) friendly.

For example, <FIG> schematically illustrates a system <NUM> including a plurality of concurrently running processes. As illustrated in <FIG>, the system <NUM> includes a client device <NUM>. The client device <NUM> is a computing device such as, for example, a desktop computer, laptop computer, a tablet, a smart phone, a smart wearable, a smart television, or the like.

The client device <NUM> includes an input-output interface <NUM>, an electronic processor <NUM>, and a memory <NUM>. The input-output interface <NUM>, the electronic processor <NUM>, and the memory <NUM> communicate wirelessly, over wired communication channels or a bus, or a combination thereof. It should be understood that the client device <NUM> may include additional components than those illustrated in <FIG> in various configurations. For example, in some embodiments, the client device <NUM> includes multiple electronic processors, multiple memory modules, multiple input-output interfaces <NUM>, and the like.

The input-output interface <NUM> allows the client device <NUM> to communicate with communication networks, peripheral devices, and the like. For example, the input-output interface <NUM> may include Ethernet ports, universal serial bus (USB) ports, and other communication ports. The electronic processor <NUM> may be a microprocessor, an application-specific integrated circuit (ASIC), and the like. The electronic processor <NUM> is generally configured to execute software instructions to perform a set of functions, including the functions described herein. The memory <NUM> includes a non-transitory computer-readable medium and stores data, including instructions that are executable by the electronic processor <NUM>. For example, as illustrated in <FIG>, the memory <NUM> stores an operating system <NUM>. The operating system <NUM> is executed by the electronic processor <NUM> to manage resources of the client device <NUM>, including managing access to locally stored data, including data stored in the memory <NUM> and other memory modules of the client device <NUM>.

For example, as described above, the operating system <NUM> may be configured to generate an index for data, such as partitioned data sets, stored in the memory <NUM>, other memory modules of the client device <NUM>, or a combination thereof. As illustrated in <FIG>, the index <NUM> may store unique keys (pointers) to stored data, metadata regarding stored data, and the like. In some embodiments, each entry in the index <NUM> is a primary key. Also, in embodiments where multiple indices are created, each index may be serialized into a separate file in the file system. This separation allows for independent operations per index without impacting other indices.

Accordingly, as described above, the index <NUM> allows processes (executed by the electronic processor <NUM>) to quickly find data, such as partitioned data sets stored on the client device <NUM>. For example, in addition to the operating system <NUM>, the client device <NUM> may store one or more software applications (referred to herein as applications <NUM>) executable by the electronic processor <NUM>. The applications <NUM> may include a productivity application (such as a word processing application, a slide presentation application), a gaming application, a media application, and/or a browser application. Each instance of these applications <NUM> executed by the electronic processor <NUM> is a process, and the electronic processor <NUM> supports concurrently running processes, wherein the processes may include multiple instances of the same application <NUM>, instances of different applications <NUM>, or a combination thereof.

As also described above, the index <NUM> must be managed to ensure data integrity while limiting delays. <FIG> is a flowchart illustrating a method <NUM> for managing the index <NUM> among a plurality of concurrently running processes on the client device <NUM> according to one embodiment. The method <NUM> is described as being performed by a process running on the client device <NUM> (as executed by the electronic processor <NUM>). The method <NUM> is performed by the operating system <NUM> as part of the file input/output (IO) stack.

As illustrated in <FIG>, the method <NUM> includes reading, with one of the plurality of processes, the index <NUM> at a first state (at block <NUM>). As described above, each entry in the index <NUM> includes a key and an associated value. The process reading the index <NUM> stores a copy of the index <NUM> that the process uses to access data. The process also maintains a journal of operations (changes) performed by the process on the index <NUM> from the first state (the last read) (at block <NUM>). Operations performed by a process with respect to the index <NUM> are limited to read operations, create operations, and delete operations. No update operations are allowed. Accordingly, any update to the index <NUM> is represented in the journal as a delete operation (to delete the existing entry to be updated) and a subsequent create operation (to add a new entry with the updated entry). Each delete operation stored in the journal specifies a key for an index entry and each create operation stored in the journal specifies a key for an index entry and an associated value for the entry.

In response to a predetermined event, the process applies the journal to the index <NUM> to update the index <NUM> (at block <NUM>). The predetermined event is a termination of a process.

Since the index <NUM> may have changed from the last time the process read the index <NUM>, the process may experience a conflict when applying the journal to the index. For example, as illustrated in <FIG>, applying the journal to the index includes reading the current state of the index <NUM> (a second state) (at block <NUM>) and sequentially (time-wise) applying the journaled operations to the current state of the index. In particular, when the journal includes a delete operation for an entry that exists in the current state of the index <NUM> (the delete operation specifies a key included in the current state of the index <NUM>), the process deletes the existing entry from the current state of the index <NUM> (at block <NUM>). Similarly, when the journal includes a create operation for an entry that does not exist in the current state of the index <NUM> (the create operation specifies a key not included in the current state of the index <NUM>), the process adds a new entry to the index <NUM> as specified by the journal (at block <NUM>).

However, when the journal includes a create operation for a new entry that already exists in the current state of the index <NUM> (the create operation specifies a key that exists in the current state of the index <NUM>), the process deletes the existing entry from the index <NUM> and adds a new entry to the index <NUM> as specified by the journal (at block <NUM>). Also, when the journal includes a delete operation for an entry that does not exist in the current state of the index <NUM> (the delete operation specifies a key that does not exist in the current state of the index <NUM>), the process ignores the delete operation included in the journal.

In some embodiments, when updating an index, there is no requirement for in-place alteration of indices. Therefore, updates to an index may be implemented using full re-writes of an index (which tend to be more SSD friendly and are naturally atomic) or in-place changes (which tend to be more friendly to rotational media). Also, each process that has interest in a particular index's data may be interested in changes to the index. Accordingly, in some embodiments, an operating system event may be used to indicate changes made to an index. Any process listening for such an event can reload (re-read) the index and update its state accordingly if needed.

<FIG> illustrates an example of using the method <NUM> to manage a plurality of processes <NUM> (including Process A, Process B, and Process C) using an index <NUM>. As illustrated in <FIG>, the index <NUM> includes at least one entry and each entry includes a key (for example, a unique numerical identifier) and an associated value (for example, a text string, a pointer, a numerical value, or the like). As illustrated in <FIG>, in a first state 305a, the index <NUM> includes a first entry (<NUM>, a) and a second entry (<NUM>, b). In a second state 305b, the index <NUM> includes the second entry (<NUM>, b) and a third entry (<NUM>, c). In a third state 305c, the index <NUM> includes the second entry (<NUM>, b) and a fourth entry (<NUM>, d). In a fourth state 305d, the index <NUM> includes the second entry (<NUM>, b) and the fourth entry (<NUM>, d).

As described above, each process in the multi-process environment that has an interest in the index <NUM> reads the current state of the index <NUM> and uses the read version of the index <NUM> to access data as needed. Also, as each process changes the index <NUM>, the process maintains a journal of such changes as either a create operation, a delete operation, or a combination thereof. As described above, instead of immediately applying changes to the index <NUM>, the journal allows a process to track changes the process will make to the index <NUM>, which allows other processes to more freely read and access the index <NUM>. In other words, by using the journal, a process does not make in-place alterations of the index <NUM> but rather applies the journal to the index <NUM> in response to predetermined events, which can be configured to balance data integrity with resource usage. Accordingly, each journal is a delta of the index <NUM> until the journal is applied.

For example, <FIG> illustrates a journal <NUM> associated with Process A tracking changes to the index <NUM> made by Process A as of the first state of the index <NUM> (the last read of the index <NUM> performed by the Process A). The journal <NUM> includes a delete operation specifying a key of "<NUM>" and a create operation specifying a key of "<NUM>" and an associated value of "c. " As illustrated in <FIG>, each delete operation added to a journal specifies a key for an entry in the index <NUM> and each create operation added to a journal specifies a key and an associated value for an entry in the index <NUM>. As noted above, no update operations are allowed. Thus, any update to the index <NUM> performed by a process is tracked in the journal as a delete operation and a create operation.

When the predetermined event occurs, the Process A applies the journal <NUM> to the current state of the index <NUM> to commit the changes to the index <NUM> made by Process A. In particular, the Process A reads the current state of the index <NUM> (the second state) and sequentially applies each operation included in the journal <NUM> to the current state of the index <NUM>. Thus, the Process A deletes the existing entry in the current state of the index <NUM> with the key of "<NUM>" and adds a new entry to the current state of the index <NUM> with a key of "<NUM>" and an associated value of the "c. " In some embodiments, after applying the journal to the current state of the index <NUM>, the Process A starts a new journal to track operations to the new current state of the index <NUM>.

As illustrated in <FIG>, the Process B also maintains a journal <NUM>, which includes a create operating specifying a key of "<NUM>" and an associated value of "d. " When the Process B applies the journal <NUM> to the current state of the index <NUM>, the index <NUM> has changed (based on the changes made by Process A) since the Process B read the index <NUM> and, thus, a conflict exists. For example, the journal <NUM> includes a create operation specifying a key of "<NUM>" and an associated value of "d," but the current state of the index <NUM> (a third state) already includes an entry with a key of "<NUM>. " Thus, when applying this create operation to the current state of the index <NUM>, the Process B deletes the existing entry from the index <NUM> with the key of "<NUM>" and adds a new entry to the index <NUM> with a key of "<NUM>" and an associated value of "d" as specified by the journal <NUM>.

Similarly, when Process C attempts to apply the journal <NUM> to the current state of the index <NUM> (a fourth state), the current state of the index <NUM> no longer includes an entry with a key of "<NUM>. " Thus, the Process C ignores the delete operation (converts to a no operation (NOP)) included in the journal <NUM> that specifies the key of "<NUM>.

Accordingly, not only do the per-process journals allows processes to track updates to the index <NUM> without have to wait for access to the index to apply the change, but by limiting the operations to delete and create operations and applying the above conflict logic, multiple processes can use an index efficiently and accurately.

In a multi-process environment, access to the index <NUM> (to read, write, or both) may be limited to one process at a time. Accordingly, the index <NUM> may be locked to all other processes whenever a process is reading or writing to the index <NUM>. However, to further reduce delays in gaining access to the index <NUM>, the operating system <NUM> may separate read-only operations from read-write operations with read-write locking semantics to provide view-serialization and associated consistent results of operations to the index <NUM>.

For example, the operating system <NUM> may be configured generate one or more mutexes for the index <NUM>. A mutex (also referred to as a mutual exclusion object) is an object that controls access to data by processes, such as limiting access to the data by no more than one process at a time. Accordingly, a mutex maintains data integrity by allowing only one process to access data at a time. Therefore, a process cannot access data that is being changed or used by another process until the other process is done, which preserves data integrity.

In some embodiments, the operating system <NUM> generates a read mutex and a write mutex for the index <NUM> to provide concurrent access by two or more processes to the index <NUM> for read purposes (to limit busy-waits for read operations) while imposing reliable waits for write operations. For example, the read mutex controls access to the index for read purposes, and the write mutex controls access to the index for write purposes. In some embodiments, any number of processes may concurrently access the index <NUM> through the read mutex. For example, when a process wants to access the index <NUM>, the read mutex may increment a counter that indicates a number of processes currently reading the data and the decrement the counter once the process has finished reading the data to track a number of processes currently reading the index. The write mutex may use this counter to determine when a write to the index <NUM> should be allowed. In other words, while a process is still reading the index <NUM> via the read mutex, the write mutex may prevent any other process from writing to the index <NUM>.

The write mutex allows a process to write (change) the index <NUM>. For example, when process needs to change data associated with a key included in the index <NUM>, such as to apply a journal, the process accesses the index <NUM> via the write mutex and updates the index <NUM> accordingly. While the process is writing to the index <NUM>, another process is locked out of writing to the index <NUM>. Accordingly, a process may experience delays when attempting to write to the index <NUM> when another process is reading or writing to the index <NUM>. However, as noted above, separating read-only requests from read-write requests through the use of the mutexes reduces overall delays for accessing the index <NUM>.

Thus, embodiments described herein provide methods and systems for managing an index shared between multiple processes through journaling for each process a delta on the existing index until the journal is applied. The use of the journals describing changes to an index allows conflicts to be scoped down to actual modifications of the index (as compare to general write or read conflicts). It should be understood that the methods and systems described herein are not limited to multi-process environments on a single computing device but may be used manage an index used by multiple processes executed on a plurality of computing devices, including, for example, computing devices operating within a cloud computing environment or the like. Similarly, the methods and systems described herein may be used at a process level, a thread level, or a combination thereof.

Claim 1:
A system configured to manage an index (<NUM>) for accessing data stored in a storage medium, the data comprising files in a file system, wherein the index maps unique keys to stored data, the index being shared by a plurality of processes, wherein an instance of an application (<NUM>) executed by the electronic processor is a process, the application comprising a word processing application, a slide presentation application, a gaming application, a media application, and/or a browser application, the system comprising:
at least one electronic processor configured to execute the plurality of processes, and
wherein the at least one electronic processor is configured to perform steps as part of a file input/output stack of the operating system (<NUM>) executed by the electronic processor, the steps comprising:
maintain (<NUM>), for each of the plurality of processes, a journal of operations to be performed on the index at a first state by each of the plurality of processes, each entry in the index including a key and an associated value and each operation included in a journal for one of the plurality of processes including a create operation or a delete operation and each operation specifying a key;
in response to a termination of the one of the plurality of processes, apply (<NUM>) the journal of the one of the plurality of processes to update the index, wherein applying the journal includes:
reading (<NUM>), with the one of the plurality of processes, the index at a second state,
deleting (<NUM>) an existing entry from the index for each delete operation included in the journal specifying a key included in an entry of the index at the second state,
adding (<NUM>) a new entry to the index for each create operation included in the journal specifying a key not included in an entry of the index at the second state,
deleting (<NUM>) an existing entry from the index and adding a new entry to the index for each create operation included in the journal specifying a key included in an entry of the index at the second state, and
ignoring (<NUM>) each delete operation included in the journal specifying a key not included in an entry of the index at the second state.