Patent Description:
Various systems and applications involve analysis of large volumes of content, such as textual items, videos, images, voice files and sensor data, to name just a few examples. Some analysis tasks use metadata that is associated with the content.

The description above is presented as a general overview of related art in this field and should not be construed as an admission that any of the information it contains constitutes prior art against the present patent application. <CIT> discloses a data storage device including a memory and a controller. The memory includes a first partition and a second partition. The controller includes a pattern detector that is configured to detect one or more tags in data from an access device to be stored in the first partition. The controller is configured to generate, in the second partition, one or more links to the data that is stored in the first partition, the one or more links organized according to metadata associated with the one or more tags.

It is the object of the present invention to provide an SSD controller which facilitates an improved analysis of unstructured media objects which are stored or are en-route to be stored on the SSD.

The object is solved by the subject matter of the independent claim which defines the present invention.

An embodiment that is described herein provides a metadata computation apparatus including a host interface, a storage interface and one or more processors. The host interface is configured to communicate over a computer network with one or more remote hosts. The storage interface is configured to communicate with one or more non-volatile memories of one or more storage devices. The one or more processors are configured to manage local storage or retrieval of media objects in the one or more non-volatile memories, to compute metadata for a plurality of the media objects that are stored, or that are en-route to be stored, on the one or more storage devices, wherein the media objects are of multiple media types, wherein the computed metadata tags a target feature in the media objects of at least two different media types among the multiple media types, and to store, in the one or more non-volatile memories, the metadata tagging the target feature found in the at least two different media types, for use by the one or more hosts.

In some embodiments, at least some of the media objects are unstructured media objects, and, in computing the metadata, the one or more processors are configured to tag locations at which the target feature appears in the unstructured media objects. In an embodiment, for a media item that includes a sequence of frames, the one or more processors are configured to tag the locations by identifying and tagging one or more of the frames in which the target feature appears. In an example embodiment, wherein, for a media item that includes at least a frame, the one or more processors are configured to tag the locations by identifying and tagging one or more coordinates in the frame in which the target feature appears. In a disclosed embodiment, the one or more processors are configured to compute the locations in accordance with multiple different location metrics defined for the respective media types.

In some embodiments, the one or more processors are configured to receive from the one or more hosts, over the computer network, one or more models that specify extraction of the metadata from the media objects, and to generate the metadata based on the received models. In an embodiment, the one or more processors are configured to receive from the one or more hosts a respective model for each of the multiple media types. In an example embodiment, the one or more processors are configured to receive, as the one or more models, one or more pre-trained Artificial Intelligence (AI) models. In a disclosed embodiment, the one or more processors are configured to generate the metadata by applying a same AI inference engine to the AI models.

In another embodiment, the one or more processors are configured to organize the media objects in multiple batches corresponding to the media types, and to compute the metadata over each of the batches. In yet another embodiment, the one or more processors are configured to generate the metadata during idle periods during which at least some resources of the one or more processors are free from managing storage of the media objects.

In still another embodiment, the one or more processors are configured to combine, in a unified metadata database, metadata that tags the target feature and that was extracted from different media sources or extracted by different processors. Additionally or alternatively, the one or more processors are configured to combine the metadata, which tags the target feature, in a unified metadata database that identifies at least one attribute selected from a group of attributes consisting of a media type, a file identifier of a file containing the media object, and a location of the media object within the file.

There is additionally provided, in accordance with an embodiment that is described herein, a method for metadata computation. The method includes communicating by a storage controller of one or more storage devices over a computer network with one or more remote hosts, and communicating with one or more non-volatile memories of the one or more storage devices. Local storage or retrieval of media objects in the one or more non-volatile memories is managed using the storage controller. Further using the storage controller, metadata is computed for a plurality of the media objects that are stored, or that are en-route to be stored, on the one or more storage devices. The media objects are of multiple media types, and the computed metadata tags a target feature in the media objects of at least two different media types among the multiple media types. The metadata, which tags the target feature found in the at least two different media types, is stored in the one or more non-volatile memories for use by the one or more hosts.

There is also provided, in accordance with an embodiment that is described herein, a metadata computation apparatus including a host interface, a storage interface and one or more processors. The host interface is configured to communicate over a computer network with one or more remote hosts. The storage interface is configured to communicate with one or more non-volatile memories of one or more storage devices. The one or more processors are configured to compute metadata for a plurality of media objects that are stored, or that are en-route to be stored, on the one or more storage devices, wherein the media objects are of multiple media types, wherein the computed metadata tags a target feature in the media objects of at least two media types among the multiple media types, and to store the metadata for use by the one or more hosts.

There is further provided, in accordance with an embodiment that is described herein, a method for metadata computation. The method includes communicating by a storage controller of one or more storage devices over a computer network with one or more remote hosts, and communicating with one or more non-volatile memories of the one or more storage devices. Using the storage controller, metadata is computed for a plurality of media objects that are stored, or that are en-route to be stored, on the one or more storage devices, wherein the media objects are of multiple media types, and wherein the computed metadata tags a target feature in the media objects of at least two of the media types, and the metadata is stored for use by the one or more hosts.

The present disclosure will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which:.

Embodiments that are described herein provide improved methods and systems for generating metadata for media objects, and for storing and using such metadata, in data processing systems.

In some embodiments, a data processing system is used for storing and analyzing a large volume of content data contained in media objects. Some non-limiting examples of object analysis applications include identification of persons of interest or other objects in video footage of security cameras, customized insertion of advertisements ("ads") into streamed videos, analytics of data from autonomous vehicles, analytics of call and response quality in a ChatBot Voice calls data base, text documents and/or text messages database analysis, mood detection, scene identification within a video file or Voice call, identification of persons or objects in surveillance footage, identification of types of actions occurring in surveillance footage, identification of voices or types of sounds in recordings, classification of phrases and/or responses used during conversation, analysis of automotive sensor data and driving responses, and many others.

Examples of media objects include videos, sound recordings, still images, textual objects such as text messages and e-mails, data obtained from various types of sensors such as automotive sensors and Internet-of Things (IoT) sensors, database objects, and/or any other suitable objects. Media objects are also referred to herein simply as "objects" for brevity.

Typically, the system generates metadata for the media objects and performs the desired analysis, or executes an action, based on the metadata. In a system for ad insertion, for example, the system would typically analyze each streamed video, divide the video into scenes, estimate a sentiment associated with each scene, people and/or objects in the scene, identify the context of speech in the scene, and store the estimated scene-sentiments and other information as metadata associated with the video. Based on this metadata, the system can then choose where in the video to insert a given ad in order to maximize the ad's impact.

As another example in an entirely different field, a data processing system is used for off-line analysis of data acquired by cameras and other sensors of an autonomous car. In this example, the system may scan the video acquired by the car cameras and/or outputs of other car sensors, identify events that are pre-classified by the AI model as being of interest, and tag them as metadata. The system can then use the metadata to investigate the identified events.

In an example embodiment, in an automotive system, extremely large quantities of sensor data are generated. Metadata is generated on the storage side of a vehicle that is periodically connected to a network. The metadata is used to select portions of relevant sensor data to be at least temporarily stored at a storage device in the vehicle, or discarded. Metadata along with selected relevant objects are then periodically uploaded over a network connection to a central processor where the metadata objects are analyzed and can be applied for various purposes such as improving the behavior of autonomous vehicles, or determining targeted advertisements that are to be conveyed to users of the vehicle.

In many cases, the media objects are unstructured. In the present context, the term "unstructured object" means that the media content of the object (e.g., textual content, audio content, image content or video content) is provided in raw form and is not organized in advance according to a fixed field format. Typically, an unstructured object is not tagged a-priori with metadata that defines any aspects of the content per frame or other content portion. Typically, unstructured data is non-transactional, and its format does not readily conform to a relational database schema.

Metadata can be generated from media objects, which are almost always unstructured, in various ways. One possibility is to use an Artificial Intelligence (AI) model, e.g., a neural network. In a typical implementation, an AI model is trained using a "training set" - a body of media objects and corresponding metadata that is known to be accurate. The trained model is then applied to generate metadata for other media objects. A software or hardware module that receives a pre-trained AI model and uses it to compute metadata of objects is referred to herein as an "AI inference engine. " In some implementations, several different AI models will be applied to unstructured or partially structured media objects.

Embodiments that are described herein provide methods and systems for generating, storing and using metadata relating to target features that are common to media objects of multiple different media types. In one illustrative example, the target feature is a person of interest. Occurrences of this target feature in media objects of different media types may comprise, for example, references to the person in database files, e-mails and/or text messages, appearances of the person's face in images, videos and/or Web pages, appearances of the person's voice in audio recordings, and the like.

In many practical applications it is highly desirable to be able to analyze occurrences of a target feature (e.g., a person) jointly across multiple media types.

In some embodiments, a data processing system comprises one or more host servers ("hosts") that analyze media obj ects, and one or more storage devices in which the media objects are stored. The hosts and the storage devices communicate over a computer network. In various embodiments, a processor in the data processing system computes metadata for a plurality of media objects that are stored, or that are en-route to be stored, on the storage devices. Among other features, the computed metadata tags a certain target feature in media objects of at least two of the multiple media types.

In an embodiment, the processor creates and maintains a unified metadata database that indexes and stores metadata of media objects of different media types, in accordance with the target feature.

For example, when the target feature is a person of interest, in some embodiments the metadata tags e-mail or other textual media objects that refer to the person, database files in which the person is listed, images and/or videos in which the person's face appears, and/or audio files in which the person's voice can be found. In an embodiment, the processor uses a common identifier to tag the target feature (the person of interest in this example) across the multiple different media types. In another embodiment the metadata is grouped by target features. Any other suitable data structure or representation can be used.

The processor typically makes the unified metadata database accessible to the hosts. In this manner, the hosts are able to retrieve and analyze metadata and objects of different media types that relate to the target feature.

In some embodiments, the metadata is indicative of the locations within the media objects in which the target feature appears. The locations may comprise, for example, a frame number or elapsed time from the beginning of a video, coordinates within a frame in a video or still image, elapsed time in an audio file, or the number of words from the beginning of a textual object.

In some embodiments the processor runs an AI inference engine, which is configured to run AI models that compute metadata for media objects. The AI model typically differs from one media type to another, e.g., the AI model for e-mail messages differs from the AI model for videos. In an example embodiment, the processor loads an AI model of a certain media type, and then computes metadata for a batch of objects of that type, before switching to an AI model of another media type. This batched mode of operation is efficient, since the AI model is replaced at large intervals.

In some embodiments, the processor that performs metadata generation is located at the storage edge, close to the locations at which the objects are stored, rather than at one centralized location which conventionally requires the transport of a very large body of media object data through a network between storage and processing devices.

In the present context, the term "at the storage edge" means at a processor or processors that are located on the storage-device side of the network, as opposed to the host-side of the network. In other words, a processor at the storage edge needs to send and/or receive information across the network in order to communicate with the hosts, but does not need to send and/or receive information across the network in order to communicate with the storage device. Non-limiting examples of storage-edge processors, as elaborated below, include Solid State Drive (SSD) controllers, processors in storage-array controllers (also referred to as aggregators), and processors in connected storage-edge appliances such as autonomous vehicles.

<FIG> is a block diagram that schematically illustrates a data processing system <NUM>, in the present example a data center, which performs metadata generation at the storage edge, in accordance with an embodiment that is described herein. System <NUM> is used for storing a large number of objects, calculating metadata for the objects, and analyzing the objects based on the metadata, as explained above. In some embodiments system <NUM> receives, stores and analyzes objects of multiple different types.

In the embodiment of <FIG>, data processing system <NUM> comprises one or more host servers <NUM> that communicate over a computer network <NUM>. Host servers <NUM> are also referred to herein simply as "hosts" for brevity. Computer network <NUM> may comprise any suitable type of network, e.g., a Local Area Network (LAN), Wide Area Network (WAN), cloud network, data center network or the like. In an embodiment, system <NUM> comprises one or more media generators <NUM> that generate the media objects being stored and analyzed.

System <NUM> further comprises a remote storage rack <NUM>, which is used by hosts <NUM> for storing objects, metadata and other relevant data. In some embodiments, storage rack <NUM> is part of a Storage Area Network (SAN) that communicates with network <NUM>. Hosts <NUM> communicate with storage rack <NUM> over network <NUM> for storing and retrieving data.

In the example of <FIG>, storage rack <NUM> comprises multiple storage units referred to as All-Flash Arrays (AFAs) <NUM>. (In alternative embodiments, any other suitable type of storage units, and any other suitable type of storage devices, not necessarily Flash-based, can be used. ) A Top-Of-Rack (TOR) switch <NUM> manages the communication between AFAs <NUM> and hosts <NUM> over network <NUM>. In the present example, storage rack <NUM> including its various components is regarded as located "at the storage edge" of system <NUM>.

An inset at the bottom-left of <FIG> shows the internal structure of AFA <NUM>, in an embodiment. As seen, AFA <NUM> comprises multiple Solid State Drives (SSDs) <NUM> in which the data (e.g., objects and metadata) is stored. AFA <NUM> comprises a storage controller <NUM>, which is configured to manage storage of data (e.g., media objects). Storage controller <NUM> is also referred to herein as an aggregation processor or aggregator. AFA <NUM> further comprises a switch <NUM> that is configured to communicate with TOR switch <NUM> over a suitable network cable <NUM>.

In some embodiments, switch <NUM> communicates with SSDs <NUM> over a common Peripheral Component Interconnect Express (PCIe) bus, e.g., using the Non-Volatile Memory Express (NVMe) protocol. In other embodiments, hosts <NUM> communicate with SSDs <NUM> via switch <NUM> using Ethernet, e.g., using the NVMe-over-fabrics protocol. Further alternatively, other suitable protocols can be used. Communication between switch <NUM> and TOR switch <NUM> is typically conducted using Ethernet. In an embodiment, although not necessarily, AFA <NUM> comprises a Central Processor Unit (CPU) and/or Network Interface Controller (NIC), not shown in the figure, for communicating with TOR switch <NUM>.

An inset at the bottom-right of the figure shows the internal structure of SSD <NUM>, in accordance with an embodiment that is described herein. In the present embodiment, each SSD <NUM> comprises a plurality of Flash memory devices <NUM>, e.g., NAND Flash memories, and an SSD controller <NUM>. SSD controller <NUM> comprises a memory interface <NUM> for communicating with Flash devices <NUM>, a host interface <NUM> for communicating with hosts <NUM> (via switch <NUM> and TOR switch <NUM>), and a processor <NUM>. Host interface <NUM> may communicate with hosts <NUM> using any suitable storage protocol, for example Non-Volatile Memory Express (NVMe) or Serial Advanced Technology Attachment (SATA).

As will be explained in detail below, processor <NUM> performs both storage/retrieval tasks and metadata computation tasks. Non-limiting examples of metadata computation tasks include identification, description and/or tagging of objects, activities, scene characteristics and other features of content within media objects. In yet other embodiments, metadata computation tasks are performed by storage controller (aggregator) <NUM>.

In an embodiment, processor <NUM> comprises a Flash management module <NUM> and an Artificial Intelligence (AI) inference engine <NUM>. Flash management module <NUM> is configured to store and retrieve data (e.g., objects and metadata) in Flash devices <NUM>. The tasks of Flash management module <NUM> are also referred to as "Flash Translation Layer" (FTL). AI inference engine <NUM> is configured to compute metadata for media objects, as explained below. In an embodiment, SSD controller <NUM> stores in Flash devices <NUM> (i) media objects <NUM> and (ii) a metadata database <NUM> that holds the metadata of media objects <NUM>.

Note that the SSD configuration of <FIG> is a non-limiting example configuration, and that any other suitable SSD controller can be used in alternative embodiments. For example, an alternative embodiment, in which the SSD controller comprises multiple Integrated Circuits (ICs) that communicate with one another via a suitable fabric, is described in <FIG> below.

<FIG> is a block diagram that schematically illustrates a data processing system <NUM> in which an edge appliance <NUM> performs local metadata generation, in accordance with an embodiment that is described herein. Edge appliance <NUM> comprises, for example, an autonomous car, a surveillance box, an IoT device, or any other suitable type of edge device, in an embodiment.

In an embodiment, edge appliance <NUM> communicates with a cloud-based data center <NUM> via a computer network, in the present example the Internet. Data center <NUM> comprises one or more hosts (not shown). Edge appliance <NUM> comprises a Central Processing Unit (CPU) cluster <NUM>, a local memory <NUM> (typically a Random Access Memory (RAM) or other volatile memory), and an SSD <NUM>. The internal structure of SSD <NUM>, in an embodiment, is shown in an inset on the right-hand side of the figure. The structure of SSD <NUM> is similar to that of SSD <NUM> of <FIG>.

The metadata in local databases <NUM> of the various SSDs <NUM> is accessible for use by hosts <NUM>. In some embodiments, a host <NUM> reads the metadata, and possibly associated objects or parts thereof, from SSD <NUM>. In an alternative embodiment, processor <NUM> of SSD controller <NUM> receives from a host <NUM> a request for certain metadata, and sends the requested metadata to the requesting host in response. In some embodiments, in addition to the requested metadata, processor <NUM> also sends to the requesting host one or more related media objects, or parts thereof. In other embodiments, processor <NUM> sends some or even all of the metadata to hosts <NUM> proactively, i.e., regardless of any request from the hosts. In an embodiment, a host <NUM> analyzes the metadata and requests selected segments of object media based on analysis of the metadata.

In some embodiments, an analysis task of a host <NUM> requires the use of metadata from multiple metadata databases <NUM> of different SSDs <NUM>. In such embodiments, the host typically obtains the relevant metadata from the multiple SSDs, and consolidates the metadata as needed.

In some embodiments, the AI model and the objects for storage are received from the same host <NUM>. In other embodiments, the AI model and the objects for storage are received from different hosts <NUM>.

In some embodiments, the data storage process and the metadata generation process are carried out concurrently. In some embodiments, processor <NUM> gives higher priority to storage/retrieval of objects than to metadata generation. As noted above, in some embodiments AI inference engine <NUM> generates the metadata during idle periods during which at least some resources of processor <NUM> are free from managing the storage/retrieval of objects. In an example embodiment, processor <NUM> identifies such idle periods in real time, and initiates or resumes metadata generation during the identified idle periods. Processor <NUM> suspends generation of metadata outside the identified idle periods. In an embodiment, while generation of metadata is suspended (outside the idle periods) processor <NUM> buffers unprocessed media objects in volatile memory until the associated metadata is generated. Alternatively, processor <NUM> may use any other suitable prioritization scheme for giving precedence to storage/retrieval over metadata generation.

<FIG> is a block diagram that schematically illustrates an SSD <NUM>, in accordance with an alternative embodiment that is described herein. In the present example, SSD <NUM> comprises multiple Flash devices <NUM>, e.g., NAND Flash memories, an SSD controller <NUM> and (optionally) a Dynamic RAM (DRAM) <NUM>. SSD controller <NUM> comprises a host interface <NUM>, a media controller <NUM>, a Static RAM (SRAM) <NUM>, one or more CPUs <NUM>, a DRAM controller <NUM>, and an AI inference engine <NUM> that optionally comprises a CPU <NUM>.

Host interface <NUM> communicates with hosts <NUM>, e.g., in accordance with the NVMe, SATA protocol or other suitable storage protocol. Media controller <NUM> is a processor that manages storage and retrieval of data in Flash devices <NUM>, similarly to Flash management module <NUM> of <FIG> and <FIG>. DRAM controller <NUM> manages storage of data in DRAM <NUM>. The various elements of SSD controller <NUM> communicate with one another via a fabric <NUM>, which typically comprises suitable data and control buses.

Among other features, the example of <FIG> demonstrates that in some embodiments the SSD controller (or other storage device controller) comprises multiple processors that jointly perform data storage/retrieval (e.g., storage/retrieval of media objects) and metadata computation. The multiple processors may reside in one or more Integrated Circuits (ICs), packaged in a single package as a single device, or in multiple separate packages.

In some embodiments, one or more processors in the data processing system generate metadata relating to a target feature that is common to media objects of multiple different types. In various embodiments, the processor or processors in question are located at the storage edge and comprise processor <NUM> in SSD controller <NUM> of system <NUM> (<FIG>) or system <NUM> (<FIG>), and/or storage controller (aggregator) <NUM> of system <NUM> (<FIG>). The description that follows refers simply to "the processor" and "the system" for clarity and brevity.

As noted above, in one example embodiment the target feature is a person of interest. In other embodiments, the disclosed techniques can be applied to any other suitable type of target feature that is found in media objects of different types. Some non-limiting examples of target features comprise place names (e.g., an airport, street name or city), companies or other organizations (e.g., company logo, name, text, Campus picture or address), events (e.g., a wedding that can be referenced in a video film, in a text invitation, a voice call or still images of bride and groom), to name just a few examples.

<FIG> is a flow chart that schematically illustrates a method for metadata generation, in accordance with an embodiment that is described herein.

Typically, the processor receives from one or more of hosts <NUM> a set of pre-trained AI models, each AI model being configured to generate metadata from objects of a respective media type. For example, the processor may receive an AI model for videos, another AI model for text messages, yet another AI model for audio files, etc. In an embodiment, at least two of the AI models (corresponding to at least two media types) are configured to identify and tag the common target feature.

Typically, the processor comprises, or has access to, a mapping that identifies the various media objects for which metadata is to be computed, their respective locations in the storage device and their respective media types. In one example embodiment, in which the processor is processor <NUM> of SSD controller <NUM>, the media objects in question are stored in Flash devices <NUM> of the SSD. In another example embodiment, in which the processor is storage controller (aggregator) <NUM> of AFA <NUM>, the media objects in question are stored in SSDs <NUM> of the AFA. A mapping of this sort may be created, for example, by a higher-layer file system or object database. In an example embodiment, the processor receives the mapping from one of hosts <NUM>, along with a command to generate the metadata.

The method begins with the processor selecting a media type for which metadata is to be generated, at a type selection operation <NUM>. At a model loading operation <NUM>, the processor loads the AI model that corresponds to the selected media type to AI inference engine <NUM>. In the present example, each media type corresponds to a single respective AI model. In alternative embodiments, however, such a one-to-one relationship is not mandatory. For example, a certain media type may be associated with several AI models, which are chosen based on other factors.

At an object identification operation <NUM>, the processor identifies (using the above-described mapping) the media objects of the currently selected media type. At a metadata generation operation <NUM>, the processor generates metadata for the media objects of the currently selected media type using the currently loaded AI model. At a database storage operation <NUM>, the processor stores the generated metadata in metadata database <NUM>. The method then loops back to type selection operation <NUM> above, in which the processor proceeds to select the next media type for metadata generation. An example process of this sort is demonstrated in <FIG> below.

Typically, the method of <FIG> is carried out continuously, e.g., as a background or low-priority task of the processor. Such continuous operation accounts for, for example, new media objects that continue to arrive for storage over time.

In some embodiments, the metadata database <NUM> created using the method of <FIG> is a unified metadata database, which jointly indexes and stores metadata of media objects of multiple different media types. Among other features, the processor tags and indexes at least some of the media objects, belonging to two or more of the media types, in accordance with the common target feature. An example format of a unified metadata database is depicted in <FIG> below.

In the example embodiment in which the target feature is a person of interest, the processor tags in the database, for example, references to the person in database files, e-mails and/or text messages, appearances of the person's face in images, videos and/or Web pages, appearances of the person's voice in audio recordings, and the like.

In an embodiment, the processor uses a common identifier to tag the target feature in the various media objects of the multiple different media types in database <NUM>. In another embodiment the processor groups the metadata in database <NUM> by target features.

Tagging the common target feature in such a unified database enables hosts <NUM> to efficiently analyze a wide variety of media objects related to the target feature, even when the media objects originate from different sources and/or are of multiple media types and/or are generated at different processors, for instance processors disposed in different storage devices at the storage edge as described in further detail in U. Patent Application entitled "SYSTEMS AND METHODS FOR GENERATING METADATA DESCRIBING UNSTRUCTURED DATA OBJECTS AT THE STORAGE EDGE" (Attorney Docket No. MP11049/<NUM><NUM><NUM>), U. Patent Application entitled "STORAGE EDGE CONTROLLER WITH A METADATA COMPUTATIONAL ENGINE" (Attorney Docket No. MP11060/<NUM><NUM><NUM>), U. Patent Application entitled "STORAGE AGGREGATOR CONTROLLER WITH METADATA COMPUTATION CONTROL" (Attorney Docket No. MP11065/<NUM><NUM><NUM>), and U. Patent Application entitled "METADATA GENERATION AT THE STORAGE EDGE" (Attorney Docket No. MP11073/<NUM><NUM>), cited above, whose disclosures are all incorporated herein by reference.

Since the unified database is generated at the storage edge, the disclosed techniques eliminate the need to transport media objects across the network for the sake of computing the metadata. Typically, most if not all of the media objects remain within the boundaries of the storage devices, and it is the metadata that is provided to the hosts for analysis. Based on the metadata, the hosts are able to select which specific media object, or portions thereof, if at all, need to be retrieved over the network. As such, traffic overhead over the computer network is reduced considerably. The disclosed techniques also reduce latency, e.g., because they require less data movement and since they enable distributed metadata generation in parallel by multiple storage-device controllers. Since the disclosed solution lends itself to distributed implementation across multiple storage devices, the solution is highly scalable. Since the disclosed technique improves metadata generation speed, it responds rapidly to updates of media objects.

In one example implementation, each entry in unified metadata database <NUM> represents an appearance of the common target feature. Each entry comprises, at least: (i) the media type of the media object, (ii) file identifier of a file in which the media object resides or will be stored, and (iii) the location of appearance of the target feature within the media object. Any other suitable data structure or representation can be used.

The processor typically expresses the location of appearance by a location metric that is suitable for the type of media. In various embodiments, the locations of appearance comprise, for example, a frame number or elapsed time from the beginning of a video, coordinates within a frame in a video or still image (e.g., coordinates of a diagonal of a rectangle surrounding the target feature), elapsed time in an audio file, the number of words from the beginning of a textual object, or any other suitable location metric.

The method flow of <FIG> is an example flow that is depicted solely for the sake of conceptual clarity. In alternative embodiments, any other suitable method can be used for metadata computation and creation of a unified metadata database. For example, the description above refers mainly to metadata generation for media objects that are already stored in non-volatile memory (NVM). Additionally or alternatively, the disclosed techniques can also be used for generating metadata for media objects that are on-the-fly, i.e., en-route to being stored in the NVM.

<FIG> is a diagram that schematically illustrates a process of producing a unified metadata database, in accordance with an embodiment that is described herein. In the example of <FIG>, the target feature is a person of interest, and the media objects being processed are stored on a certain SSD.

The left-hand side of <FIG> shows multiple media objects of multiple media types intermixed on the SSD. In this embodiment, objects marked "V" are video objects, objects marked "A" are audio objects, and objects marked "T" are textual objects.

The middle of <FIG> shows the stage at which the media objects are batched by media type, ready for batch processing by the AI inference engine. As seen, video objects, audio objects and textual objects are batched separately. Appearances of the common target feature (references to the person of interest in this example) within the objects are marked "x". As seen, the target feature appears in objects of multiple media types. Some objects comprise multiple occurrences of the target feature, other objects comprise a single occurrence, and in yet other objects the target feature does not appear at all.

The right-hand side of <FIG> shows the unified metadata database created by the processor at the storage edge, e.g., by processor <NUM> of SSD controller <NUM> or by aggregator <NUM>. Each entry in the unified metadata database corresponds to a respective occurrence of the common target feature. Each entry specifies attributes such as the media type of the media object ("V", "A" or "T" in the present example), a unique identifier of the file including for example the file name and its storage location, and the location of the occurrence of the target feature within the file.

The system and storage device configurations depicted in <FIG> above, and the database configuration depicted in <FIG> above, are example configurations, which were chosen solely for the sake of conceptual clarity. <FIG> show example configurations in which a controller of a storage device (e.g., an SSD controller in an SSD or an aggregator of an AFA) communicates over a computer network with one or more remote hosts, and locally with the NVM of the storage device. The controller comprises one or more processors that jointly compute and store in the NVM metadata of media objects that are stored, or that are to be stored, in the NVM. In alternative embodiments, any other suitable system and/or storage-device configuration can be used. Elements that are not mandatory for understanding of the disclosed techniques have been omitted from the figures for the sake of clarity.

In alternative embodiments, the disclosed techniques can be used with other suitable types of storage devices, e.g., Hard Disk Drives (HDDs) in which the storage medium is magnetic.

The various elements of data processing systems <NUM> and <NUM>, and of their components such as SSDs <NUM> and <NUM> and AFA <NUM> and its components, as well as SSD <NUM> and its components, e.g., AI inference engine <NUM>, may be implemented using dedicated hardware or firmware, such as using hard-wired or programmable logic, e.g., in an Application-Specific Integrated Circuit (ASICs) or Field-Programmable Gate Array (FPGA), using software, or using a combination of hardware and software elements.

Typically, processor <NUM> of SSD controller <NUM>, aggregator <NUM>, CPU <NUM> and/or CPU <NUM> comprise programmable processors, which are programmed in software to carry out the functions described herein (e.g., Flash management and metadata computation). The software may be downloaded to the processor in electronic form, over a network, for example, or it may, alternatively or additionally, be provided and/or stored on non-transitory tangible media, such as magnetic, optical, or electronic memory.

Although the embodiments described herein mainly address media processing, the methods and systems described herein can also be used in other applications, such as in behavior analytics of people.

Although the embodiments described herein mainly address metadata generation at the storage edge, the methods and systems described herein can also be used for metadata generation by a processor or processors located on the host-side of the network. Hybrid solutions, in which metadata generation tasks are partitioned between processors at the storage edge and on the host side, are also feasible.

Claim 1:
A Solid State Drive, SSD, controller (<NUM>) comprised by an SSD (<NUM>), the SSD controller (<NUM>) comprising:
a host interface (<NUM>), configured to communicate over a computer network with one or more remote hosts;
a storage interface (<NUM>), configured to communicate with one or more non-volatile memories (<NUM>) of the SSD (<NUM>); and
one or more processors (<NUM>), configured to:
manage local storage or retrieval of unstructured media objects (<NUM>) in the one or more non-volatile memories (<NUM>);
compute metadata for a plurality of the unstructured media objects (<NUM>) of multiple different media types that are stored, or that are en-route to be stored, on the SSD (<NUM>), wherein the plurality of unstructured media objects (<NUM>) of multiple different media types comprises video, audio, still images, text, data obtained from various types of sensors, wherein the computed metadata tags a target feature in the unstructured media objects (<NUM>) of multiple different media types of at least two different media types among the multiple media types; and
store, in the one or more non-volatile memories (<NUM>), the computed metadata tagging the target feature found in the at least two different media types, for use by the one or more hosts,
wherein the one or more processors (<NUM>) are configured to combine the computed metadata, which tags the target feature, in a unified metadata database (<NUM>) stored in the one or more non-volatile memories (<NUM>), wherein each entry in the unified metadata database (<NUM>) represents an appearance of the target feature in one of the unstructured media objects (<NUM>) of multiple different media types and comprises at least the media type of the respective unstructured media object (<NUM>) of multiple different media types and a location of appearance of the target feature within the respective unstructured media object (<NUM>) of multiple different media types.