Patent ID: 12261996

While the above-identified figures set forth one or more embodiments of the present disclosure, other embodiments are also contemplated, as noted in the discussion. In all cases, this disclosure presents the invention by way of representation and not limitation. It should be understood that numerous other modifications and embodiments can be devised by those skilled in the art, which fall within the scope and spirit of the principles of the invention. The figures may not be drawn to scale, and applications and embodiments of the present invention may include features and components not specifically shown in the drawings.

DETAILED DESCRIPTION

FIG.1is a schematic of system10for selection/extraction, preprocessing, and publishing to subscribers of video data of a region of interest (i.e., a scene) that is a subset of a first field of view of the streaming video. System10can include camera12, configuration file14, gateway/container16, and publishing location/endpoint18. Camera12can include streaming video data20having entire frame22with a first field of view. Scenes24A,24B, and24C (i.e., individual regions of interest) can be selected/extracted from entire frame22each having a second field of view, a third field of view, and a fourth field of view, respectively, that are less than the first field of view of entire frame22. Camera12collects streaming video data20and transfers/sends streaming video data20to gateway/container16. Gateway/container16preprocesses streaming video data20according to preprocessing parameters defined in configuration file14and publishes (i.e., allows access/makes available) the preprocessed video data as first video data21A (for scene24A), second video data21B (for scene24B), and third video data21C (for scene24C) to publishing location/endpoint18. Subscribers26A-26D can subscribe to video data21A-21A of each scene24A-24C located at endpoint18to access each scene24A-24C.

System10can include machine-readable storage media. In some examples, a machine-readable storage medium can include a non-transitory medium. The term “non-transitory” can indicate that the storage medium is not embodied in a carrier wave or a propagated signal. In certain examples, a non-transitory storage medium can store data that can, over time, change (e.g., in RAM or cache). In some examples, storage media can be entirely or in part a temporary memory, meaning that a primary purpose storage media is not long-term storage. Storage media, in some examples, is described as volatile memory, meaning that the memory, does not maintain stored contents when power to system10(or the component(s) where storage media are located) is turned off. Examples of volatile memories can include random access memories (RAM), dynamic random-access memories (DRAM), static random-access memories (SRAM), and other forms of volatile memories. In some examples, storage media can also include one or more machine-readable storage media. Storage media can be configured to store larger amounts of information than volatile memory. Storage media can further be configured for long-term storage of information. In some examples, storage media include non-volatile storage elements. Examples of such non-volatile storage elements can include magnetic hard discs, optical discs, flash memories and other forms of solid-state memory, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. Most generally, storage media is machine-readable data storage capable of housing stored data from a stored data archive.

System10can also include one or multiple computer/data processors. In general, the computer/data processors can include any or more than one of a processor, a microprocessor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other equivalent discrete or integrated logic circuitry. System10can include other components not expressly disclosed herein but that are suitable for performing the functions of system10and associated methods of preprocessing and processing video data and other forms of data. For example, system10can include communication software and/or hardware for pushing/sending configuration file14from storage media at a location distant from gateway/container16, such as cloud storage, to gateway/container16for execution/implementation on streaming video data20.

Camera12can be any device capable of collecting streaming video data20, such as a Real Time Streaming Protocol (RTSP) camera or a USB camera. Streaming video data20can be video data that is continuously captured/recorded by camera12in any suitable format. Camera12can be positioned/located to provide streaming video data20displaying entire frame22with a first field of view. The first field of view shown/displayed by camera12can be a wide field of view that shows multiple regions of interest. Video data20being collected, for example, can show a wide field of view of a warehouse for storing commercial products and/or an assembly line producing commercial products of which each individual subscriber26A-26D may only be interested in a region/scene24A-24C that is a subset of entire frame22. Camera12can collect and transfer streaming video data20in any resolution/video quality and any format, including (but not limited to) MP4, AVI, FLV, WMV, MOV, MPEG, Motion JPEG, AVCHD, WebM, and/or MKV. Camera12can transfer/send streaming video data20to gateway/container16over any suitable means, including via the internet, short-range wireless technology, or any other type of wired and/or wireless connection.

Configuration file14is an editable file that contains preprocessing parameters that define, among other instructions, how streaming video data20is to be preprocessed by gateway/container16to create video data21A-21C. Configuration file14can include numerous other instructions for gateway/container16, including which camera12to connect to (i.e., receive streaming video data20from), what portion of entire frame22to select/extract to create scenes24A-24C (i.e., how to crop frame22to create scenes24A-24C), and at which endpoint18to publish the preprocessed scenes24A-24C. This is discussed in detail with regards toFIG.2. Configuration file14can be edited/revised and pushed/conveyed to gateway/container16for execution in real time (i.e., runtime) such that an editor can revise the preprocessing parameters and those revisions can be applied to scenes24A-24C at runtime. Configuration file14can be an executable program file or have another format for including instructions and conveying information that is then used by gateway/container16to apply the preprocessing to video streaming data20. Additionally, configuration file14can be stored in storage media adjacent to and/or part of gateway/container16or in storage media distant from gateway/container16, such as in the cloud. Configuration file14can be accessible only by one editor or can be accessible by multiple parties (e.g., editors), which may include subscribers26A-26D who can edit the preprocessing parameters defined by/within configuration file14to instruct gateway/container16to preprocess one or each of scenes24A-24C depending on the needs/desires of subscribers26A-26D, respectively.

Gateway/container16can include a computer processor capable of performing instructions provided by configuration file14, which can include preprocessing parameters that are to be applied to streaming video data20. Gateway/container16can be a gateway node, edge device, container, virtual machine, or other software and/or hardware able to accept configuration file14and perform the instructions therein to apply the preprocessing parameters to streaming video data20. Further, gateway/container16can be within a single computer hardware set up due to virtualization. Gateway/container16can include one or multiple storage media for storing information, such as the preprocessing parameters pushed/sent to gateway/container16by/from configuration file14and/or other information like streaming video data20. Gateway/container16can be located at various locations, including adjacent to camera12and/or on the same network as camera12, distant from camera12with streaming video data20being received by gateway/container16from camera12via a wired or wireless connection, in the cloud, or at multiple locations. Gateway/container16is in communication with configuration file14to accept instructions for applying preprocessing parameters. Additionally, gateway/container16may be configured to contact configuration file14to determine if configuration file14has been edited. If edited, gateway/container16can perform preprocessing (on streaming video data20being received) according to the newly edited configuration file14. In other examples, gateway/container16can utilize preprocessing parameters or other information included within configuration file14on a period basis, and can utilize changes (or edits) made to configuration file14when encountered in a next iteration of the period.

For clarity, streaming video data20is unaltered video data that is received by gateway/container16from camera12, whereas first video data21A is video data that has been preprocessed by gateway/container16(according to preprocessing parameters defined in configuration file14) to create scene24A. Similarly, second video data21B is video data that has been preprocessed to create scene24B, and third video data21C is video data that has been preprocessed to create scene24C. For each of first, second, and third video data21A-21C, the originating video data is streaming video data20(which can be converted to raw video data20A as described with regards toFIG.3below). Each of scenes24A-24C can be a subset of entire frame22and show second, third, and fourth fields of view, respectively, that are less than the first field of view of entire frame22. However, other scenes can have the same field of view as first field of view of entire frame22and instead, other editing is performed on that scene besides cropping; for example, that scene can be edited to be in grayscale whereas entire frame22of streaming video data20is in color.

Gateway/container16can be in communication with endpoint18to which gateway/container16publishes the preprocessed video data21A-21C (e.g., scenes24A-24C). The communication can be wired or wireless, such as communication via the internet. However, endpoint18can be at the same location as gateway/container16or on the same computer hardware set up and/or network. Further, endpoint18can be located on the internet with a unique address and/or security protocol that allows for subscription and access to scenes24A,24B, and24C. Scenes24A,24B, and24C can be published to endpoint18using an asynchronous messaging library, for example ZeroMQ, such that scenes24A,24B, and24C are published as topic1, topic2, and topic3, respectively. Subscribers26A-26D can subscribe to any of topics1-3to receive video data21A-21C of scenes24A,24B,24C, respectively. Gateway/container16can publish video data21A-21C of each of scenes24A-24C to endpoint18in any format suitable for use by subscribers26A-26D. For example, video data21A-21C can each be published as Motion JPEG or any of the formats listed above with regards to streaming video data20. The format that video data21A-21C of each of scenes24A-24C can be designated in configuration file14and applied to video data21A-21C by gateway/container16.

Each scene24A-24C at topics1-3, respectively, can be subscribed to by any number of subscribers26A-26D. In the example shown inFIG.1, scene24A has two subscribers26A and26B, while scene24B has one subscriber26C, and scene24C has one subscriber26D. Video data21A-21C of each of scenes24A-24C can be further processed by subscriber26A-26D, respectively, depending on the desired output/inference to be determined from video data21A-21C. For example, first video data21A of scene24A can be further processed by an AI model to determine the amount of a particular product that has passed by camera12(in the second field of view of scene24A) on an assembly line. This is described in detail with regards toFIG.5.

FIG.2is a schematic of configuration file14. Each “component” of configuration file14can be one or a set of instructions that, when executed by gateway/container16, applies a process/edit. Thus, while this disclosure may discuss the components of configuration file14as being physical, tangible elements, the components can be one or multiple groups of executable software code contained within configuration file14. Configuration file14includes information regarding camera credentials28and preprocessing parameters30A-30C corresponding to scenes24A-24C. Preprocessing parameters30B and30C can include the same type of information/parameters as preprocessing parameters30A. However, for simplicity, the possibilities of information/parameters shown inFIG.2with regards to preprocessing parameters30A are not shown for preprocessing parameters30B and30C. However, preprocessing parameters30B and30C can be the same or different from each other and from preprocessing parameters30A corresponding to scene24A. Preprocessing parameters30A can include topic name/publishing location32, video format34, accompanying information36, and preprocessing pipeline38having various video edits38A-380. Video edits38A-380can include the nonexclusive list of crop38A, grayscale38B, contrast38C, brightness38D, threshold38E, resize38F, blur38G, hue saturation value (HSV)38H, sharpen381, erosion38J, dilation38K, Laplacian image processing38L, Sobel image processing38M, pyramid up38N, and pyramid down380.

Configuration file14can be edited and pushed/conveyed to gateway/container16in real time (i.e., runtime) such that preprocessing parameters30A-30C (and the other information contained in configuration file14) can be applied to streaming video data20immediately to preprocess and output video data21A-21C. Configuration file14can be stored, edited, and/or pushed/conveyed to gateway/container16in any suitable format/file type, such as a text file, a comma separated value file, or other format/file type. Configuration file14can include other information/parameters not expressly disclosed herein and not shown inFIG.2.

Configuration file14can include camera credentials28, which provides the information needed for gateway/container16to connect to camera12and/or receive streaming video data20from camera12. Camera credentials28can include other information such as encryption/decryption information, security access information, and/or instructions for beginning and/or ending the collection of streaming video data20by camera12. Camera credentials28can include information for connecting to multiple cameras12and/or information for gateway/container16to receive the same or different streaming video data20from the same or different cameras12for different scenes24A-24C. In one example, camera credentials28are provided once and applied to all scenes24A-24C. In another example, different camera credentials28are provided for each scene24A-24C and applied to each of scenes24A-24C individually.

Configuration file14also includes information specific to each scene24A-24C. The information/instructions are designated as preprocessing parameters30A,30B, and30C, respectively. Preprocessing parameters30A-30C are used by gateway/container16(e.g., in the form of executable instructions or indications of the executable instructions) and applied to streaming video data20to create video data21A-21C of scenes24A-24C, respectively. Preprocessing parameters30A-30C can include topic name/publishing location32, which designates where video data21A-21C of scenes24A-24C will be published after preprocessing. As described below with regards to gateway/container16andFIG.3, publishing location32can be a proxy location which is then relayed to unified endpoint18to make locating the topic/scenes24A-24C easier for subscribers26A-26D. Publishing location32can be any other location suitable for providing access to subscribers26A-26D.

Configuration file14can designate video format34that each of scenes24A-24C is to be published at, which can be the same format as streaming video data20or any other type of suitable video format, including the formats listed above with regards to streaming video data20and/or video data21A-21C. Scenes24A-24C can be published having the same video format34or different video formats34. If the format of the video data of scenes24A-24C is to be changed, the steps of changing the video format can be performed before, during, or after any of the other instructions/steps set out in preprocessing parameters30A-30C. For example, video format34can be changed before, during, or after video edits38A-380are performed by gateway/container16.

Preprocessing parameters30A-30C can also include accompanying information36, which is information provided/published with video data21A-21C for each of scenes24A-24C. Accompanying information36can include any information about first video data21that may be of use to subscribers26A-26C. For example, accompanying information36can include first video data21A frame size, which may be helpful in indicating to subscriber26A what processing should be performed on first video data21A of scene24A; if the frame size is 720 pixels by 486 lines, first video data21A of scene24A may be most suitable for processing by an AI model. Accompanying information36can include metadata and/or other information regarding what preprocessing has been performed on streaming video data20to create video data21A-21C for scenes24A-24C, respectively.

Preprocessing parameters30A-30C can also include preprocessing pipeline38that includes numerous video edits38A-380that can be applied to streaming video data20to create video data21A-21C for each of scenes24A-24C. Preprocessing pipeline38can designate the instructions for the entire video edits made to streaming video data20for each of scenes24A-24C. The amount of time gateway/container16takes to perform the video edits designated by preprocessing pipeline38(i.e., video edits38A-380) for each of scenes24A-24C can be measured, recorded, and displayed as preprocessing performance metrics (seeFIG.4). The order in which video edits38A-380are performed can be optimized (or otherwise enhanced for increasing accuracy, decreasing bandwidth or latency, etc.) by gateway/container16and/or optimized within configuration file14to reduce the amount of time gateway/container16takes to perform preprocessing pipeline38. This optimization can be done manually by an editor (e.g., the individual with access to edit configuration file14) or automatically by system10. For example, depending on which video edits38A-380are to be performed in preprocessing pipeline38by gateway/container16, the order of performance of those video edits38A-380can be rearranged to reduce the amount of time gateway/container16takes to perform preprocessing pipeline38. In one example, crop38A is the first video edit38A-380to be performed, followed by other video edits38B-380.

Video edits38A-380are a nonexclusive list of edits that can be designated in configuration file14and performed on streaming video data20by gateway/container16. Preprocessing pipeline38can include other video edits not expressly included in the list of video edits38A-380. Similarly, not all of video edits38A-380need to be performed to create video data21A-21C of each of scenes24A-24C, and different scenes24A-24C can include different video edits38A-380performed on streaming video data20by gateway/container16. In one example, only crop38A is performed on streaming video data20to create first video data21A of scene24A, while a different crop38A and brightness38D, resize38E, and dilation38K are performed on streaming video data20to create second video data21B of scene24B that is different than first video data21A of scene24A.

Each of video edits38A-380are briefly described as follows. Crop38A is the removal of unnecessary areas/regions (i.e., regions that are not of-interest to the subscriber) of entire frame22having first field of view to create scenes24A-24C each with second, third, and fourth field of views, respectively. Scenes24A-24C that have been cropped38A have fields of view that are a subset of (i.e., less than) first field of view of entire frame22. Grayscale38B is the alteration of the color of video data21A-21C and can include limiting the color to be between white and black. Contrast38C is the alteration of the difference between the maximum and minimum pixel intensity. Brightness38D is the alteration in the luminous brightness of video data21A-21C. Threshold38E is the alteration of the color of video data21A-21C by changing the color of select pixels of video data21A-21C that are above a specified threshold color value. Resize38F is the alteration of the frame size of video data21A-21C without cutting/cropping any of the frame out. Blur38G is the alteration of the clarity of video data21A-21C, which may be desired for some processing applications, such as an AI model, performed by subscribers26A-26D. Hue saturation value (HSV)38H is assigning a numerical readout of video data21A-21C that corresponds to the color contained therein. Sharpen381is altering video data21A-21C to make the objects therein appear more defined/sharpened. Erosion38J is altering video data21A-21C by shrinking pixels and/or removing pixels on object boundaries, while dilution38K is the reverse of erosion in that video data21A-21C is enlarged by resizing pixels and/or adding pixels at object boundaries. Laplacian image processing38L and Sobel image processing38M are processing techniques known in the art that can be applied to video data21A-21C. Pyramid up38N and pyramid down380are altering video data21A-21C by smoothing and/or subsampling as known in the art. Each of scenes24A-24C can include these and other video edits38A-380to be applied by gateway/container16to preprocess streaming video data20and output as scenes24A-24C for use by subscribers26A-26D.

Configuration file14can arrange the instructions of camera credentials28and preprocessing parameters30A-30C to be performed in any order, or gateway/container16can have the capabilities to arrange/rearrange the information/instructions to be performed in a desired/optimized sequence. Additionally, gateway/container16can be configured to perform each set of preprocessing parameters30A-30C in parallel such that preprocessing parameters30A, preprocessing parameters30B, and preprocessing parameters30C are performed at the same time (and the time gateway/container16takes to perform those preprocessing parameters are measured, recorded, and displayed as metrics). Configuration file14can be edited at any time by an editor and then pushed/conveyed/accessed by gateway/container16at runtime such that the preprocessing of video data21A-21C is altered according to the newly edited configuration file14at runtime.

FIG.3is a schematic of gateway/container16(hereinafter, “gateway16”) along with inputs to and outputs from gateway16. Each “component” of gateway16(and corresponding inputs and outputs) can be one or a set of instructions, programs, processors, storage media locations, and/or other software or hardware used to select/extract, preprocess, and publish video data21A-21C as scenes24A-24C. Thus, while this disclosure may discuss the components of gateway16(and corresponding inputs and output) as being physical, tangible elements, the components can be partially or entirely contained within software and/or hardware.

Inputs to gateway16can be streaming video data20(received from camera12) and configuration file14, which includes camera credentials28and preprocessing parameters30A-30C. Outputs from gateway16can be scenes24A-24C to unified endpoint18, which is the location at which scenes24A-24C are published as topics1-3, and metrics40, which is the location at which preprocessing pipeline38information (i.e., the amount of time gateway16takes to apply preprocessing parameters30A-30C to streaming video data20to create scenes24A-24C) is published/accessible. Gateway16can include raw video data20A, which is streaming video data20that has been published at internal topic42and to which gateway16subscribes to receive video data used to create each of scenes24A-24C. Gateway16can include preprocessing pipeline optimization and preprocessing44, which uses preprocessing parameters30A-30C as defined in configuration file14to preprocess streaming video data20(accessed as raw video data20A at internal topic42) to create scenes24A-24C. Gateway16publishes scenes24A-24C to proxy location46at topics1-3, respectively. Gateway16can then publish/relay scenes24A-24C (having video data21A-21C) from proxy location46to unified endpoint18.

Gateway16receives streaming video data20from camera12and can publish the video data as raw video data20A at internal topic42. This configuration provides for a constant, known location of published raw video data20A independent of where the original streaming video data20is received from. Thus, if the location where streaming video data20is being received from changes (e.g., if one camera is disconnected but another camera comes online), raw video20A will still be accessible at internal topic42without the need to change where gateway16is looking for video data to create scenes24A-24C, thus ensuring a smooth transition of incoming streaming video data20. Raw video data20A published at internal topic42can be configured such that only gateway16has access.

Gateway16can be provided with the information in configuration file14(i.e., camera credentials28and preprocessing parameters30A-30C) via a variety of avenues. In one example, gateway16has location information of configuration file14and actively accesses configuration file14. In another example, configuration file14is pushed/conveyed to gateway16once, periodically, or continuously and gateway16passively waits to begin preprocessing streaming video data20until configuration file14has been received. Another example can be a combination of the two above examples in that gateway16actively accesses configuration file14at the beginning of preprocessing (and continues preprocessing following those known instructions) and configuration file14is pushed/conveyed to gateway16only after configuration file14has been edited/revised.

Gateway16can perform pipeline optimization on preprocessing parameters30A-30C As discussed above, pipeline optimization can be performed by gateway16(or another component) to reduce the time gateway16takes to preprocess raw video data20A to create video data21A-21C (and attach any additional information) of scenes24A-24C and/or to increase accuracy of the preprocessing operations. Pipeline optimization can include arranging/rearranging the order in which video edits38A-380are performed by gateway16.

Gateway16can then, according to preprocessing parameters30A-30C defined within configuration file14, preprocess44raw video data20A (which is derived from and similar to streaming video data20) to edit raw video data20A to create video data21A-21C of scenes24A-24C. The preprocessing to create each of scenes24A-24C can be performed individually for each scene and can be performed in parallel (i.e., simultaneously). The preprocessing performed by gateway16can edit each of scenes24A-24C to the desires/needs of subscribers26A-26C, respectively. For example, gateway16can preprocess raw video data20A to crop a first field of view of entire frame22to eliminate areas/regions of the first field of view that are of no interest to subscriber26A and keep a second field of view of, for example, scene24A that is a subset of the first field of view. Thus, further processing by subscriber26A (or subscriber26B) does not need to be performed on the areas (regions not of-interest) eliminated/trimmed by the cropping performed during preprocessing. In this example, the cropping reduces the processing time and resources needed by subscriber26A. The alteration of video format can be included in the preprocessing and/or can be performed before, during, or after the application of other preprocessing parameters30A-30C.

Scenes24A-24C, which are made up of raw video data20A that has been preprocessed according to configuration file14to create video data21A-21C, are published/sent to proxy location44. Because scenes24A-24C are continuous video data, scenes24A-24C are continuously published (e.g., made available) to subscribers26A-26C, respectively. Scenes24A-24C are published/sent first to proxy location44and then relayed to unified endpoint18. Having scenes24A-24C first being published/sent to proxy location44ensures that, no matter what path scenes24A-24C take, video data21A-21C of scenes24A-24C will always end up at proxy location44and then be relayed to unified endpoint18. Because scenes24A-24C always end up at proxy location44, unified endpoint18always knows the location to access scenes24A-24C and can actively look to proxy location44to obtain scenes24A-24C or passively wait for scenes24A-24C to be published/sent to unified endpoint18.

During preprocessing to create scenes24A-24C, the amount of time gateway16takes to apply all of preprocessing parameters30A-30C to raw video data20A to create scenes24A-24C, respectively, (or the amount of time gateway16takes to apply only video edits38A-380of preprocessing pipeline38, depending on the desired measurement) is measured, recorded, and published at metrics40for viewing by an editor or any party with access. Metrics40can be published on an asynchronous messaging library like ZeroMQ (similar to the publishing of scenes24A-24C) or can be displayed on a user interface similar toFIG.4, which shows a display of scene preprocessing performance metrics40A. Metrics40can be outputted from gateway16to numerous other systems and/or locations, including to the cloud or another location distant from gateway16.

As shown inFIG.4, scene preprocessing performance metrics40A show a value of time for each of scenes24A-24C. This value of time displayed is the amount of time gateway16takes to apply preprocessing parameters30A-30C for each of scenes24A-24C. Alternatively, this value of time displayed can be the amount of time gateway16takes to apply only video edits38A-380of preprocessing pipeline38and not the entirety of preprocessing parameters30A-30C. This measurement may be desired because the other instructions/information of preprocessing parameters30A-30C (other than video edits38A-380) may not change between scenes24A-24C while preprocessing pipeline38(the video edits38A-380being applied) may be drastically different from scene to scene and thus the amount of time for performing preprocessing pipeline38may vary greatly from scene to scene.

For example, preprocessing performance metrics40A inFIG.4show an amount of time gateway16takes to perform the video edits of preprocessing pipeline38for scene24A as 5.2 milliseconds (ms), for scene24B as 9.4 ms, and for scene24C as 3.3 ms. In this example, it may be desirable or necessary to alter preprocessing parameters30B in configuration file14(either remove some edit and/or rearrange the order in which those edits are performed by gateway16) to reduce the amount of time gateway16takes to apply preprocessing pipeline38to create/alter scene24B. As discussed above, gateway16can have the capability to optimize the order in which the edits/instructions in preprocessing parameters30A-30C and/or preprocessing pipelines38are performed to reduce the amount of time needed to preprocess/apply the edits/instructions. Thus, preprocessing performance metrics40A as shown inFIG.4may be the shortest amount of time gateway16takes to perform preprocessing parameters30A-30C (or preprocessing pipeline38, depending on the chosen measuring points).

Scenes24A-24C can be published from proxy location44to unified endpoint18as topics1-3within an asynchronous messaging library, such as ZeroMQ. Unified endpoint18allows for a known, constant location to which subscribers26A-26C can look to access scenes24A-24C, respectively. If more scenes are created, those scenes would be published and accessible at unified endpoint18, so subscribers26A-26D and/or other subscribers would know where to look to access the additional scenes.

FIG.5is a schematic showing processing48capable of being performed on scene24A by first subscriber26A. Processing48can also be performed on scenes24B and24C or other scenes not disclosed herein.

After being published/sent to endpoint18, scenes24A-24C are available to be subscribed to and accessed by subscribers26A-26D. Each scene can be subscribed to by any number of subscribers as is necessary/desirable. For example, a scene may be subscribed to by numerous subscribers each running processing48that includes different AI models. In this example, one AI model can be determining the amount of a first product that is passing through the scene on an assembly line, while a second AI model can be determining the amount of a second product that is passing through the scene on the same assembly line. In this case, the scene is unchanged between the two AI models (i.e., between the two subscribers) but the processing performed by each subscriber after the scene has been published is different. It should be noted that one entity (e.g., a person, company, quality control sector) can subscribe to a scene multiple times and thus constitute multiple subscribers. As discussed above, the preprocessing performed to create/alter each scene can be tailored to the needs of the subscriber(s) to, for example, reduce processing48resources and time needed to determine at least one output that is indicative of an inference the subscriber is aiming to ascertain.

The example inFIG.5shows first subscriber26A performing processing48on scene24A. Processing48, as selected and executed by subscriber26A (either automatically and/or manually by a computer processor and/or other hardware and software), can include AI model48A, optical character recognition (OCR)48B, video clipping48C, further formatting48D, and display48E of the video data of scene24A. Processing48can include other instructions/edits not expressly disclosed inFIG.5and listed above.

The disclosed potential instructions/edits that subscriber26A can perform in processing48are as follows. AI model48A can be a program/model that may have machine learning and can use scene24A to determine at least one output indicative of an inference dependent upon scene24A. The inference, for example, can be the amount of a specific product that is viewable in scene24A over a defined period of time. AI model48A can also be, for example, a program/model that determines how many people appear in scene24A over a defined period of time. AI model48A can include other capabilities and/or configurations. OCR48B can be a program (or other configuration) that recognizes and records any characters (i.e., text) that appear in scene24A. For example, scene24A can be video data of a street and OCR48B will recognize and record any text that appears on the side of a vehicle, such as a delivery truck, that is in scene24A. Video clipping48C can clip the video data shown in scene24A to create a clip of a defined period of time, and/or video clipping48C can clip the video data shown in scene24A to create a static image of a defined moment in time. Further formatting48D can be video edits, such as video edits38A-380in configuration file14, or any other video or file formatting that are performed by subscriber26A. For example, further formatting48D can include cropping scene24A to be a subset of the second field of view shown in scene24A. Display48E can be making scene24A viewable on a screen or other visual display. Display48E can also include any video formatting/reconfiguring that is necessary to effectuate the display of scene24A. While video edits38A-380and further formatting48D to create and/or process video data21A-21C have included only edits to the video/image, edits can include editing the audio or other aspects of the video data.

The potential instructions/edits (i.e., processing48) can be performed in parallel or series. Further, processing48can be configured such that instructions/edits48A-48E work together such that one instruction/edit is prompted by an inference from another instruction/edit. For example, video clipping48C can be configured to work in tandem with another edit/process; if AI model48A determines that a product is defective, video clipping48C can be prompted to record and clip a particular duration (or moment to create a static image) of scene24A showing the defective product and save the clip (or image) for proof/validation.

System10, with associated methods, for selection/extraction, preprocessing, and publishing of streaming video data20into scenes24A-24C and for processing scenes24A-24C is used to determine at least one output that is indicative of an inference dependent upon video data21A-21C. System10reduces the preprocessing/processing time and resources necessary for accomplishing the desired output/determining the selected inference. System10allows for a streamlined process that extends from collecting streaming video data20from camera12to making preprocessing scenes24A-24C available to subscribers26A-26C for further processing and analysis/determinations. System10also allows for measurement, recordation, and viewing of preprocessing performance metrics40and optimization of preprocessing pipeline38(and/or preprocessing parameters30A-30C) to reduce the amount of time and resources needed to apply preprocessing parameters30A-30C to streaming video data20(i.e., raw video data20A) to create video data21A-21C of scenes24A-24C.

FIG.6is a schematic of a preprocessing and processing system including a testing system for improving a main output of a main AI model.FIG.6shows main preprocessing and processing system10(also referred to herein as first or main system/pipeline10) as well as testing system210(also referred to herein as second or testing system/pipeline210).

Main system10is similar in configuration and capabilities as the system described with regards toFIGS.1-5. As shown inFIG.6, main system10includes camera12that collects and transmits streaming video data to internal topic42(which can be within gateway16). Internal topic42can be a location at which the streaming video data is sent/published as raw video data20. Raw video data20is then accessed by gateway16and preprocessed44A to create video data21having scene24A, which can be a subset of entire frame22as shown inFIG.1. Once preprocessed to create video data21, video data21can then be published at unified endpoint18for access by subscriber26. Subscriber26can then process48video data21using main AI model48A to determine main output50indicative of a main inference dependent upon video data21. Similarly named and numbered components of system10inFIGS.1-5and main system10inFIG.6have the same configurations, functionalities, and capabilities. For additional information on these components, refer to the above disclosure with regards toFIGS.1-5.

Testing system/pipeline210can have all the components and capabilities of system10, and can include additional component and capabilities described herein. For example, testing system210can include machine-readable storage medium and/or one or multiple hardware/computer/data processors. The term “processor” as used in this disclosure can include one processor or multiple processors for performing executable instructions as described. The performance of the executable instructions can be by different processors distant from one another, by multiple processors working together, or by one processor executing all instructions. Additionally, components of similar name and/or reference number of system10and testing system210(while adding a “2” to the front of reference numbers of testing system210) can have similar configurations and/or capabilities. For example, main AI model48A of system10can be similar in configuration and capabilities to test AI model248A of testing system210. Similarly, configuration file14of system10can be similar in configuration, capabilities, etc. as test configuration file214of testing system210.

Testing system210accesses raw video data20collected by camera12and potentially published at internal topic42. Raw video data20is test preprocessed244A by testing system210to create test video data221having scene224A, which can be a subset of entire frame22as shown inFIG.1and can be the same field of view as scene24A of video data21. After raw video data20is test preprocessed244A using test preprocessing parameters defined within test configuration file214to create test video data221, test video data221is then published at test unified endpoint218for access by test AI model248A (which can be one and the same with main AI model48A). Test AI model248A processes248test video data221to determine test output250indicative of a test inference dependent upon test video data221. Parameter optimization module252can receive/access main output50(from main AI model48A), test output250(from test AI model248A), and/or potentially other information to determine if test output250satisfies the baseline criterion. If so, main configuration file14is altered so that preprocessing parameters of main configuration file14are similar to test preprocessing parameters of test configuration file214. Then, main system/pipeline10preprocesses44A the raw video data20to create video data21that is eventually analyzed/processed48by main AI model48A to produce main output50that is improved (e.g., increased correct identification of inferences, increased accuracy, and decreased latency) as compared to the previous main output50before the alteration of main configuration file14.

If test output250does not satisfy the baseline criterion, parameter optimization module252can alter the test preprocessing parameters in test configuration file214. The new, altered test preprocessing parameters of test configuration file214are then reapplied to new incoming raw video data20. New test preprocessing244A is performed to create new test video data221, which is then analyzed/processed by test AI model248A to create new test output250. Parameter optimization module252then compares new test output250to the baseline criterion. If parameter optimization module252determines that the new test output250satisfies the baseline criterion, parameter optimization module252can alter the main preprocessing parameters in main configuration file14of system10to be similar to the new test preprocessing parameters of the new test configuration file214. If new test output250does not satisfy the baseline criterion, the test preprocessing parameters are altered and the testing loop is performed again. This testing loop can continue until the test output satisfies the baseline criterion.

To initially select the test preprocessing parameters defined within test configuration file214(i.e., the test preprocessing parameters that are first applied to raw video data20at the beginning of the testing loop), parameter optimization module252can use the preprocessing parameters defined in main configuration file14as a baseline and alter those preprocessing parameters according to starting values254. As discussed below, starting values254can be selected by a user, determined from previous data, and/or be determined through other methods. The components of testing system210are described in greater detail below.

Testing system210can include machine-readable storage media and/or can include one or multiple hardware/computer/data processors. Testing system210can include other components not expressly disclosed herein but that are suitable for performing the functions of testing system210and associated methods of optimizing the preprocessing parameters to create video data21to improve the analysis/processing by main AI model48A. For example, testing system210can include communication software and/or hardware for pushing/sending optimal preprocessing parameters to main configuration file14for use in system10.

Test configuration file214is similar in configuration, functionality, and capabilities to configuration file14described with regards toFIGS.1-5(particularlyFIG.2). Test configuration file214is an editable file that contains test preprocessing parameters that define, among other instructions, how raw video data20is to be preprocessed244A to create test video data221. Configuration file214can include numerous other instructions. Test configuration file214can be editable by parameter optimization module252either initially (before the first preprocessing244A) or in response to the analysis of test output250to better format/edit test video data221for processing by test AI model248A to create improved test output250(as compared to previous iterations of test output250). Test configuration file214contains test preprocessing parameters, which can include one or multiple test video edits (see the description with regards toFIG.2for more information). Parameter optimization module252can alter anything in test configuration file214, including the test video edits of test preprocessing parameters. For example, parameter optimization module252can alter which test video edits are applied (i.e., the number and what kinds of test video edits are applied), the order in which the test video edits are applied, and the value of each of the test video edits (e.g., whether the brightness test video edit should be a value of 10 or 70).

Test configuration file214is applied to raw video data20(i.e., raw video data20is preprocessed244A) to create test video data221having test scene224A. Test scene224A can have the same field of view as scene24A of main system/pipeline10so that processing by main AI model48A and by test AI model248A can have the same baseline for analysis in creating main output50and test output250, respectively.

After applying test preprocessing parameters defined within test configuration file214to raw video data20to create test video data221, test video data221can be published to test unified endpoint218. Test unified endpoint218can be the same in configuration, capabilities, and functionality as unified endpoint18of system10, and even could be the same component so as to be one in the same with unified endpoint18such that video data21and test video data221are published at the same location to allow access by main AI model48A and/or test IA model248A. In addition to unified endpoint218, test video data221can be published/sent first to a proxy location (such as proxy location44) and then relayed to unified endpoint218.

After being published/sent to unified endpoint218, test video data221(with test scene224A) is available to be subscribed to and accessed by test AI model248A. Test AI model248A processes248test video data221to determine at least one test output250indicative of at least one test inference dependent upon test video data221. Test AI model248A and main AI model48can be similar in configuration, capabilities, and functionality and may even be the same component. Main AI model48A and test AI model248A should process video data21and test video data221, respectively, in a similar manner to ensure that the independent variables in determining main output and test output (i.e., the differences between main system/pipeline10and testing system/pipeline21) are only the main preprocessing parameters (in configuration file14) and the test preprocessing parameters (in test configuration file214) as opposed to any differences in the way main AI model48A and test AI model248A perform the processing. Main AI model48A and test AI model248A can be in communication with one another so that if any changes in the processing/analysis are made to main AI model48A, those changes are also applied to test AI model248A.

Main AI model48A and/or test AI model248A can be a program/model that may have machine learning and can use the video data to determine at least one main output50or test output250indicative of an inference dependent upon the video data. The inference, for example, can be the amount of a specific product that is viewable in scene24A or test scene224A over a defined period of time. Main AI model48A and/or test AI model248A can also be, for example, a program/model that determines how many people appear in scene24A or test scene224A over a defined period of time. Main AI model48A and/or test AI model248A can include other capabilities and/or configurations.

Main output50and test output250can each include a variety of information dependent upon video data21and test video data221, respectively. That information can include a value representative of a number of correctly identified inferences (e.g., a number of correctly identified products), a value representative of the accuracy of main output50or test output250(e.g., a number between 0 and 1 showing a probability that the identified inferences are correct), and/or a value representative of the latency of the processing by main AI model48A and test AI model248A to identify inferences (e.g., an amount of time the AI model takes to analyze the video data and identify an inference). Main output50can be accessed by and/or sent to subscriber26for further analysis, use, and/or recordation. Additionally, main output50and test output250can be accessed by and/or sent to parameter optimization module252for use in determining if and to what extent to alter test preprocessing parameters in test configuration file214.

Parameter optimization module252can include a computer processor (e.g., computer hardware and/or executable software code) capable of receiving information regarding the baseline criterion, main output50, test output250, staring values254, and other information and determining if and to what extent to alter test preprocessing parameters defined within test configuration file214and/or to alter the preprocessing parameters defined within configuration file14of main system/pipeline10. Thus, parameter optimization module252can be in wired or wireless communication with any components of main system/pipeline10and/or testing system/pipeline210. Parameter optimization module252can be similar in configuration, capabilities, and functionality as gateway16, and can be the same component as gateway16such that any computing/processing/execution by parameter optimization module214can be performed by gateway16. Parameter optimization module252can include one or multiple storage media for storing information, such as the test preprocessing parameters and/or test configuration file214, the baseline criterion, starting values254, main output50, test output250, and/or other information. Parameter optimization module252can be located at various locations, including within, adjacent to, or distant from gateway16, main AI model48A, test AI model248A, and/or any other components of main system/pipeline10and testing system/pipeline210.

Testing system/pipeline210can include a component similar to gateway16to perform all of the functions of gateway16with regards to main system/pipeline10, such as preprocessing244A raw video data20according to test preprocessing parameters in test configuration file214to create test video data221. Alternatively, preprocessing244A can be performed by parameter optimization module252such that parameter optimization module252can be in communication with internal topic42to receive/access raw video data20and in communication with test unified endpoint218to which test video data221is sent/published for access by test AI model248A.

Parameter optimization module252receives test output250(and main output50) and compares test output250to the baseline criterion. The comparison of test output250to the baseline criterion can be a comparison of any data included in test output250(such as values representative of correctly identified inferences, accuracy, and/or latency) to the baseline criterion.

The baseline criterion can be established/set through various methods. The baseline criterion can be a value or values designated by a user. For example, a user can select an accuracy value of 0.95 for the baseline criterion such that test output250must have an accuracy value at or above 0.95 to satisfy the baseline criterion. In another example, the user can select a latency value to be 10 millisecond for the baseline criterion such that test output250must have a latency value at or below 10 millisecond to satisfy the baseline criterion. In another example, the user can select a value of correctly identified inferences to be 12 per minute for the baseline criterion such that test output250must have a value of correctly identified inferences to be at or more than 12 per minute to satisfy the baseline criterion. The baseline criterion can include one or more than one of these designated values such that test output250must satisfy each of the designated baseline values to satisfy the baseline criterion. Additionally, the baseline criterion can be set by parameter optimization module252depending on a variety of factors, including the values of a previous test output250.

Moreover, the baseline criterion can be set by the respective values of main output50. In this situation, test output250is, for all intents and purposes, compared to main output50because main output50establishes the baseline criterion. For example, main output50can have an accuracy value of 0.90, which is then set as the baseline criterion. Thus, the baseline criterion is satisfied when a value representative of accuracy of test output250is greater than the value representative of the accuracy of main output50, which is 0.90. In another example, main output50can have a latency value of 15 milliseconds (i.e., a time it takes main AI model48A to process video data21to determine main output50), which is then set as the baseline criterion. Thus, the baseline criterion is satisfied when a value representative of latency of test output250(i.e., a time it takes test AI model248A to process test video data221to determine test output25) is less than the value representative of latency of main output50, which is 15 milliseconds. In another example, main output50can have a value of correctly identified inferences of 10 per minute, which is then set as the baseline criterion. Thus, the baseline criterion is satisfied when a value representative of correctly identified inferences by test AI model248A (as set out in test output250) is greater than the value representative of correctly identified inferences by main AI model248A (as set out in main output50), which is 10 per minute. The baseline criterion can include one or more than one of these values based on main output50such that test output250must satisfy each of the designated baseline values to satisfy the baseline criterion.

As mentioned above, parameter optimization module252compares test output250to the baseline criterion. If test output50satisfies the baseline criterion, parameter optimization module252can alter the preprocessing parameters in main configuration file14to be the same as the test preprocessing parameters in test configuration file214so that main output50will be similar to test output250having the improved output (after preprocessing and processing of video data by main system/pipeline10). Thus, main output50is improved/optimized through the testing performed by testing system/pipeline210. After the preprocessing parameters defined in main configuration file14are updated by parameter optimization module252, testing system/pipeline210can continue the testing loop to find even more improved/optimal preprocessing parameters.

If test output250fails to satisfy the baseline criterion, parameter optimization module252can alter the test preprocessing parameters in test configuration file214. Then, the testing loop is repeated by preprocessing244A raw video data20according to the newly altered test preprocessing parameters in newly altered test configuration file214, creating new test video data221, publishing/sending new test video data221to test unified endpoint218, and accessing and processing248A by test AI model248A to create new test output250. Parameter optimization module252then compares new test output250to the same or a different baseline criterion. This testing loop can be repeated as many times as desired, such as until test output250satisfies the baseline criterion or even after test output250has satisfied the baseline criterion and the main preprocessing parameters have been altered to be the same as the most current test preprocessing parameters.

Similar to main preprocessing parameters of main configuration file14, test preprocessing parameters of test configuration file214can include multiple test video edits (see the discussion regardingFIG.2and video edits38A-380). Parameter optimization module252can alter each and every test video edits of the test preprocessing parameters defined within test configuration file214. The number of test video edits can be altered to be greater than or less than previous test video edits. For example, newly altered test preprocessing parameters can include seven test video edits, whereas the previous test preprocessing parameters only included three test video edits. Additionally, a value of each test video edit can be altered so as to be different than previous test video edits. For example, newly altered test preprocessing parameters can include a test video edit with a brightness value of 58, whereas the previous test preprocessing parameters included a test video edit with a brightness value of 14. Moreover, an order in which the test video edits are applied to raw video data20can be altered so as to be different than an order in which the previous test video edits were applied. For example, newly altered test preprocessing parameters can be ordered such that the crop video edit is performed before the resize video edit, whereas the previous test preprocessing parameters were ordered so that the resize video edit was performed before the crop video edit. These and other alterations can be made to test preprocessing parameters in test configuration file214by parameter optimization module252in response to information received, such as main output50, test output250, and other information.

Starting values254are the initial values of the test preprocessing parameters of test configuration file214. Starting values254can be determined through various methods, including by being selected by a user and/or determined from previous data. For example, starting values254can be dependent upon the time of day at which testing system/pipeline210first begins preprocessing244A raw video data20. Additionally, starting values254can depend on starting values used for previous testing system/pipeline210operations. For example, if a previous operation of testing system/pipeline210has starting values254that returned optimal test output250, similar starting values254may be selected for use in the current operation. Other methods not expressly disclosed herein can be used to select starting values254.

Parameter optimization module252can alter the test preprocessing parameters in test configuration module252(either after test output250fails to satisfy the baseline criterion or in an attempt to further improve/optimize main output50) through various methods, including trial and error, fuzzy logic, or other method. Using trial and error, parameter optimization module252can alter one aspect of the test video edits (whether each video edit is applied, the value of each video edit, and/or the order in which the video edits are performed/applied) and determine whether test output250is closer to or further away from satisfying the baseline criterion. The trial and error can continue until the baseline criterion is satisfied. Using fuzzy logic, parameter optimization module252can alter the amount of change of the test video edits and determine to what degree test output250is closer to or further away from satisfying the baseline criterion. Depending upon that degree, parameter optimization module252will increase or decrease the amount of change of the test video edits on the next iteration. Other methods of altering the test preprocessing parameters from one testing loop to the next can be used, including a combination of the two discussed above. For any of the methods of altering the test preprocessing parameters, any number of testing loops can be performed, including performing hundreds or thousands of testing loops with different test video edits of the test preprocessing parameters.

Testing system/pipeline210provides for a method of improving main output50of AI model48A processing video data21by emulating main system/pipeline10(preprocessing44A and processing48of video data20/21) and altering the test preprocessing parameters defined within test configuration file214to determine test preprocessing parameters that return test output250that satisfies the baseline criterion. The test preprocessing parameters that return test output250that satisfies the baseline criterion are then applied to the main preprocessing parameters defined within main configuration file14of main system/pipeline10so that main AI model48A returns main output50that are an improvement upon previous main outputs50(i.e., pre-altered main configuration file14). The testing loop can continue until the preprocessing parameters are optimal and no other iterations of preprocessing parameters would return improved results of main output50.

While the invention has been described with reference to an exemplary embodiment(s), it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the essential scope thereof. Therefore, it is intended that the invention not be limited to the particular embodiment(s) disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.