Vision intelligence management for electronic devices

One embodiment provides a method comprising classifying one or more objects present in an input comprising visual data by executing a first set of models associated with a domain on the input. Each model corresponds to an object category. Each model is trained to generate a visual classifier result relating to a corresponding object category in the input with an associated confidence value indicative of accuracy of the visual classifier result. The method further comprises aggregating a first set of visual classifier results based on confidence value associated with each visual classifier result of each model of the first set of models. At least one other model is selectable for execution on the input based on the aggregated first set of visual classifier results for additional classification of the objects. One or more visual classifier results are returned to an application running on an electronic device for display.

TECHNICAL FIELD

One or more embodiments relate generally to vision intelligence (VI), and in particular VI management for electronic devices.

BACKGROUND

A large number of recognition models (e.g., deep learning models) may be available to an electronic device for use in visual analysis of visual data (e.g., performing object recognition on photos and/or videos). However, due to limited computation resources on the electronic device, it may not be feasible for all models available to the electronic device to be active simultaneously during run-time.

SUMMARY

One embodiment provides a method comprising classifying one or more objects present in an input comprising visual data by executing a first set of models associated with a domain on the input. Each model of the first set of models corresponds to an object category. Each model is trained to generate a visual classifier result relating to a corresponding object category in the input with an associated confidence value indicative of accuracy of the visual classifier result. The method further comprises aggregating a first set of visual classifier results based on confidence value associated with each visual classifier result of each model of the first set of models. At least one other model is selectable for execution on the input based on the aggregated first set of visual classifier results for additional classification of the one or more objects. One or more visual classifier results are returned to an application running on an electronic device for display.

These and other features, aspects and advantages of the one or more embodiments will become understood with reference to the following description, appended claims and accompanying figures.

DETAILED DESCRIPTION

One or more embodiments relate generally to vision intelligence, and in particular, vision intelligence management for electronic devices. One embodiment provides a method comprising classifying one or more objects present in an input comprising visual data by executing a first set of models associated with a domain on the input. Each model of the first set of models corresponds to an object category. Each model is trained to generate a visual classifier result relating to a corresponding object category in the input with an associated confidence value indicative of accuracy of the visual classifier result. The method further comprises aggregating a first set of visual classifier results based on confidence value associated with each visual classifier result of each model of the first set of models. At least one other model is selectable for execution on the input based on the aggregated first set of visual classifier results for additional classification of the one or more objects. One or more visual classifier results are returned to an application running on an electronic device for display.

For expository purposes, the term “visual data” as used herein generally refers to visual content, such as images, videos, or any other type of visual content that is displayable on a display device (e.g., a television, a monitor, tablet, smartphone, etc.).

For expository purposes, the term “vision processing task” generally refers to a task/operation involving visual analysis of visual data. Examples of different types of vision processing tasks include, but are not limited to, face recognition, scene recognition, object recognition, object localization and segmentation, object tracking, 3D object reconstruction, etc.

With the introduction of new artificial intelligence and computer vision technology in recent years, a camera application and/or a gallery application on an electronic device is evolving into an intelligent tool that, as illustrated in various embodiments, may be used to extract information from the visual domain and provide rich visual context to one or more other applications and services on the electronic device. While the camera application and/or the gallery application are mentioned in one or more embodiments for illustration, ordinary people skilled in the art would appreciate that other types of applications may equally benefit from features of the present invention.

To perform a vision processing task, a camera application on an electronic device may be extended with visual processing functionalities that are based on deep learning (DL) and neural networks. One or more embodiments provide a computer vision (CV) and DL based vision intelligence (VI) framework. The VI framework is an extension and integration framework for VI, enabling deployment of new DL and CV technologies on electronic devices with minimal application change. For example, the VI framework enables different autonomous VI components (e.g., native and/or third party/partner augmented reality applications) to run within a camera application and/or a gallery application on an electronic device, thereby extending visual processing functionalities of the camera application and/or the gallery application. Each VI component has a small footprint, enabling simple integration with the camera application and/or the gallery application. The VI framework extends visual processing functionalities of a mobile camera or other types of sensing circuits of the electronic device, enabling the mobile camera or the other types of sensing circuits to become an intelligent camera or sensing circuits instead of just a tool for capturing visual data. For example, the mobile camera may become a source of VI if the VI components support one or more of the following visual processing functionalities: (1) object recognition (i.e., identifying different types of objects in a visual field, such as a car), (2) determining attributes and relationships between objects (e.g., in response to identifying a sports car and a road in a visual field, determining that the sports car is being driven on the road), and (3) recognizing attributes of particular objects (e.g., a car belonging to a particular individual).

In one embodiment, the VI components form an open architecture engendering a component ecosystem in which different communities (e.g., research communities), device vendors, users, third parties or partners may provide deep learning models associated with visual processing functionalities.

In one embodiment, the VI components are sandboxed to isolate the camera application and/or the gallery application from one or more failures of the VI components.

In one embodiment, the VI framework enables load balancing opportunities for power optimization and resource utilization that allow the VI components to benefit from new forms of software acceleration or hardware acceleration (e.g., GPU, FPGA, special purpose neural net processing units) without changes to the camera application and/or the gallery application, thereby facilitating a high-degree of modularization.

In one embodiment, the VI framework provides support for both on-device and on-cloud components and hybrid architectures that combine on-device and on-cloud processing of visual data.

As DL and CV technologies are rapidly changing, the VI framework allows rapid integration of current and emerging vision technologies with mobile applications and services. Each VI component may be dynamically added, removed, activated, deactivated, or upgraded.

FIG. 1illustrates an example computing architecture10for implementing a VI management system300, in one or more embodiments. The computing architecture10comprises an electronic device100including computation resources, such as one or more processors110and one or more storage units120. One or more applications250may execute/operate on the electronic device100utilizing the computation resources of the electronic device100.

Examples of an electronic device100include, but are not limited to, a mobile electronic device such as a tablet, a smart phone, a laptop, a smart watch, etc.

In one embodiment, the electronic device100comprises a camera140integrated in or coupled to the electronic device100. One or more applications on the electronic device100may utilize the camera140to capture visual data (e.g., photos, video) of an object presented to the camera140.

In one embodiment, the one or more applications250on the electronic device100include, but are not limited to, one or more visual data acquiring applications and/or one or more components acquiring VI. Examples of different visual data acquiring applications include, but are not limited to, a camera application161(FIG. 4) for capturing visual data utilizing the camera140, a gallery application162(FIG. 4) for offline storage of visual data captured by the camera140or received from other sources. Examples of different components for VI include, but are not limited to, an object recognition application for classifying one or more objects present in an image/video frame, a scene recognition application for classifying one or more scenes present in an image/video frame, a car recognition application for classifying one or more cars present in an image/video frame, a third party application for customizing an online service offered by a third party based on visual data (e.g., a shopping application for providing online shopping recommendations based on one or more objects present in an image/video frame, a nutrition/diet application for providing nutrition/diet information based on one or more food items present in an image/video frame, a store/restaurant locator application for providing store/restaurant recommendations based on one or more food items present in an image/video frame, etc.), an object localization and segmentation application for detecting where one or more objects present in an image/video frame are located, an object tracking application for object tracking one or more objects over a series of image/video frames, etc.

In one embodiment, an application250may be pre-loaded onto or downloaded to the electronic device100. An application250may originate from a device vendor (e.g., an original equipment manufacturer (OEM)) of the electronic device100(i.e., a vendor application) or originate from a third party (e.g., a community, a user, a partner).

As described in detail later herein, the VI management system300provides a set of APIs and components for VI that allow application developers to enhance existing applications250and build new applications250that leverage processing of and understanding of visual data. The VI management system300provides an extension and integration framework for VI that enables one or more components for VI to run within one or more visual data acquiring applications to extend visual processing functionalities of the visual data acquiring applications, such as object recognition or face recognition.

In one embodiment, the electronic device100may further include one or more additional sensors150other than the camera140, such as, but not limited to, one or more position sensors (e.g., GPS) for determining a location of a user of the electronic device100, one or more audio sensors (e.g., microphone) for detecting user vocalizations (i.e., speech) and features of audio environment (e.g., noisy, quiet, etc.), one or more speech recognition systems for detecting spoken words, one or more text analysis systems for determining user sentiments (e.g., angry, happy, etc.), interests and intent, and one or more data analysis engines for determining information about user preferences and intent based on data collected on the electronic device100(e.g., application usage, search history, contacts, etc.). A sensor150may be utilized by an application250to capture sensor-based contextual information.

In one embodiment, the electronic device100comprises one or more input/output (I/O) units130integrated in or coupled to the electronic device100, such as a keyboard, a keypad, a touch interface, or a display screen.

In one embodiment, the electronic device100is configured to exchange data with one or more remote servers200or remote electronic devices over a connection (e.g., a wireless connection such as a WiFi connection or a cellular data connection, a wired connection, or a combination of the two). For example, a remote server200may be an online platform for hosting one or more online services (e.g., an image and video hosting website) and/or distributing one or more applications (e.g., an application250).

FIG. 2illustrates an example application250hosting a presentation layer255, in one or more embodiments. A module260is a software abstraction configured to perform a particular vision processing task (e.g., object recognition, object localization and segmentation, scene recognition, car recognition, object tracking, etc.). In an embodiment, a module260is configured to: (1) receive, as input, visual data comprising one or more image/video frames, process the visual data based on a visual processing algorithm (e.g., an object recognition algorithm), and generate a corresponding visual classifier result (i.e., recognition result) indicative of an outcome of the processing (e.g., a classification label for an object present in the visual data, such as “dog”). A module260may also rely on additional inputs, such as sensor-based contextual information captured by one or more sensors150of the electronic device100, or device information associated with the electronic device100.

An application250may utilize one or more modules260to perform one or more desired vision processing tasks. To allow simple integration of one or more modules260with an application250, in one embodiment, the VI management system300provides the application250with one or more presentation layers255to host. Each presentation layer255is a plug-in mechanism comprising a small, embeddable application configured to handle communication with one or more modules260running in one or more module containers270.

In one embodiment, a presentation layer255comprises an easy-to-use API for configuration and state management of a module260with which a hosting application250will communicate via the presentation layer255. Once the module260is configured via the presentation layer255, the hosting application250may register a listener for receiving a visual classifier result returned from the module260. The module260may return the visual classifier result to either the presentation layer255or directly to the hosting application250itself.

In one embodiment, a presentation layer255includes a user interface (UI) element and associated application logic for utilizing a module260to process visual data, and displaying a visual classifier result returned by the module260after processing.

In one embodiment, a module260may be configured for a video streaming operation or a single shot operation. A video streaming operation involves a visual data acquiring application, such as the camera application161or the gallery application162, continuously passing live image/video frames or recorded image/video frames to a module260as input for processing. A single shot operation involves a visual data acquiring application, such as the camera application161or the gallery application162, passing a single image/video frame to a module260as input for processing.

In one embodiment, one or more modules260are autonomous, such that each module260runs in its own module container270. This has the benefit of reducing launch time of the hosting application250and also ensures that the hosting application250remains running should the one or more modules260fail. Further, as a module260may have significant resource demands (e.g., significant memory and/or processing requirements), sandboxing the one or more modules260is important to preserve integrity and reliability of the hosting application250, particularly if the hosting application250is a visual data acquiring application such as the camera application161or the gallery application162.

In one embodiment, isolating a hosting application250from one or more modules260is achieved by running the one or more modules260as separate processes. For example, if the hosting application250utilizes multiple modules260, the hosting application250hosts multiple presentation layers255for launching separate processes for the modules260and communicating with the modules260through inter-process communication (IPC) and shared memory. With the presentation layers255, the hosting application250need not deal with the modules260directly and can easily turn on or off different modules260.

In one embodiment, a presentation layer255may enable a lightweight application by isolating one or more Augmented Reality (AR) modules260from the camera application161. This enables dynamic or selective addition, removal, activation, deactivation, or upgrade of AR modules260in relation to the camera application161. For example, if a view of an office is captured by the camera application161, a module260for object recognition may classify one or more furniture items present, and a different module260for scene recognition may classify the scene present as an office space.

A presentation layer255has a small footprint (i.e., small size). As such, a presentation layer255has little impact on resource utilization of a hosting application250.

FIG. 3illustrates example module families400, in one or more embodiments. In one embodiment, the VI management system300is configured to group available modules260into different module families400. Each module family400comprises a set of modules260with similar visual processing functionality. For example, a module family400may comprise modules260associated with a specific problem area, such as object tracking. As another example, a module family400may comprises modules260associated with a particular technology, such as DL.

As shown inFIG. 3, in one embodiment, the different module families400may include, but are not limited to, one or more of the following: (1) a DL module family410for DL, (2) a Face module family420for facial analysis, (3) a Gesture module family430for gesture analysis, (4) a Symbol module family440for symbol analysis, (5) a Track module family450for object tracking, (6) a Cloud module family460for cloud services, and (6) a Test module family470for testing.

In one embodiment, the DL module family410comprises, but is not limited to, one or more of the following modules260: (1) a Classification module411for classifying one or more objects present in an image/video frame, (2) a Detection module412for detecting one or more objects present in an image/video frame, (3) a Feature Extraction module413for extracting one or more features from an image/video frame, (4) a Depth Estimation module414for determining one or more measurements (e.g., distances) relating to one or more objects present in an image/video frame, (5) an Image Segmentation module415for segmenting an image/video frame into multiple segments, (6) a Style Transfer module416for recomposing an image/video frame in a style of another image/video frame (i.e., applying look and feel of one image/video frame to a different image/video frame), and (7) an Object Reconstruction module417for capturing a shape and appearance of one or more objects present in an image/video frame (e.g., generating a three-dimensional model of an object).

In one embodiment, the Face module family420comprises, but is not limited to, one or more of the following modules260: (1) a Face Detection module421for detecting a face present in an image/video frame, (2) a Face Recognition module422for recognizing/identifying a face present in an image/video frame, (3) a Face Clustering module423for measuring similarity among faces present in multiple image/video frames, and clustering similar faces into groups, and (4) an Emotion/Age/Gender module424for determining at least one of a facial expression of a face present in an image/video frame, an age of the face, or a gender of the face.

In one embodiment, the Gesture module family430comprises, but is not limited to, one or more of the following modules260: (1) a Gaze Object tracking module431for object tracking an eye gaze of an individual present in an image/video frame, (2) a Hand Gestures module432for detecting and recognizing hand gestures exhibited by an individual present in an image/video frame, and (3) a Body Features/Postures module433for detecting and recognizing at least one of body features and body postures exhibited by an individual present in an image/video frame.

In one embodiment, the Symbol module family440comprises, but is not limited to, one or more of the following modules260: (1) a Text module441for detecting and recognizing text in visual data, (2) a Handwriting module442for detecting and recognizing handwriting in visual data, and (3) a Symbol/Signs module443for detecting and recognizing at least one of symbols and signs in visual data.

In one embodiment, the Track module family450comprises, but is not limited to, one or more of the following modules260: (1) a Frame-based DL Object tracking module451for frame-based DL object tracking of one or more objects over a series of image/video frames, (2) an Optical Flow module452for performing optical flow over a series of image/video frames, (3) a pattern-based Object tracking module453for tracking patterns over a series of image/video frames, (4) a Feature Descriptor Models module454for detecting and recognizing features over a series of image/video frames, and (5) a simultaneous localization and mapping (SLAM) module455for performing SLAM over a series of image/video frames.

In one embodiment, the Cloud module family460comprises, but is not limited to, one or more modules260that bridge to one or more web-based vision services (e.g., a Microsoft® service, a Baidu® service, etc.). For example, the Cloud module family460may comprise an Amazon® module461for performing a vision processing task in association with one or more Amazon® services (e.g., providing online shopping recommendations based on one or more objects present in an image/video frame). As another example, the Cloud module family460may comprise a Google® module462for performing a vision processing task in association with one or more Google® services (e.g., online search results based on one or more objects present in an image/video frame). As another example, the Cloud module family460may comprise an IBM®/Watson® module463for performing a vision processing task in association with one or more IBM®/Watson® services (e.g., AI services).

In one embodiment, the Test module family470comprises, but is not limited to, one or more of the following modules260: (1) a Java/Java Native Interface (JNI) module471for testing Java/JNI code, and (2) a Timing module472for testing execution time of one or more processes.

FIG. 4illustrates an example module260in detail, in one or more embodiments. In one embodiment, a module260has an internal structure that includes one or more models320, one or more engines340, and an engine selector330. The module260utilizes the one or more models320for performing a particular vision processing task. Each model320comprises a dataset. In one embodiment, each model320is a neural network trained for a specific task (e.g., an image processing task, such as face recognition).

In one embodiment, the module260comprises a model ecosystem310maintaining different types of models320available for the module260to utilize. The model ecosystem310provides run-time binding of the module260with one or more models320, and high-level configuration APIs that allow provisioning of different types of models320from different sources. For example, the model ecosystem310may include, but are not limited to, one or more of the following: (1) one or more community models321, wherein each community model321is developed by a research community and is freely available, (2) one or more vendor models322, wherein each vendor model322is developed by a device vendor (e.g., an OEM) of the electronic device100and is only available to run on electronic devices from the device vendor or licensed by the vendor, (3) one or more user models323, wherein each user model323is developed based on user data (e.g., a user model323trained based on a user's collection of images/videos to identify friends or family members of the user), and (4) one or more third party/partner models324, wherein each third party/partner model324is developed by a third party and is available through partnership or licensing.

The module260is configured to receive, as input, visual data from a visual data acquiring application on the electronic device100, such as one or more camera images/videos from the camera application161or one or more gallery images/videos from the gallery application162. The module260is also configured to receive one or more additional inputs, such as sensor-based contextual information from one or more sensors150of the electronic device100, or device information associated with the electronic device100.

In one embodiment, the module260comprises a model compression unit261configured to compress a model320utilized by the module260. In one embodiment, the module260comprises an optimization unit262configured to optimize power optimization and resource utilization for load balancing. In one embodiment, the module260comprises a software acceleration unit263configured to determine whether the model320benefits from software acceleration (e.g., single instruction, multiple data (SIMD), Open Multi-Processing (OpenMP), etc.). In one embodiment, the module260comprises a hardware acceleration unit264configured to determine whether the model320benefits from hardware acceleration (e.g., central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), etc.).

In one embodiment, the module260comprises an engine selector330configured to select an engine340from a collection of available engines340(e.g., Engine1, Engine2, . . . , and Engine N) for running the module260. Each available engine340comprises a software activity running, or capable of running, the module260. The engine selector330provides load balancing and resource management for hosting applications250with multiple modules260and/or multiple models320.

In one embodiment, each model320may be supported by an engine340. Each engine340may support one or more models320.

In one embodiment, an application developer of the module260may control which engines340are associated with the module260based on requirements and/or parameters of a model320utilized by the module260. In one embodiment, the engine selector330is configured to select an appropriate engine340from a collection of available engines340that an application developer of the module260has associated with the module260.

In another embodiment, the engine selector330is configured to automatically select one or more engines340to associate with a module260. Specifically, the engine selector330is configured to: (1) dynamically determine one or more associations between the module260and one or more engines340of a collection of available engines340based on run-time data, and (2) select an appropriate engine340from the collection of available engines340for running the module260based on the one or more associations determined. The ability to dynamically determine an association between the module260and an engine340removes the requirement for a pre-determined fixed association between the module260and the engine340(e.g., a fixed association provided by an application developer of the module260). As described in detail later herein, this flexibility allows the module260to run on different types of engines340, such as third party engines, based on information such as ontology, specifics of the module260, contextual information relating to operation of the module260, etc.

After running the module260utilizing a model320on an engine340selected by the engine selector330, a visual classifier result generated by the module260may be forwarded to one or more presentation layers255hosted in one or more applications250or directly to the one or more applications250. For example, the visual classifier result may be forwarded to one or more of the following: (1) a visual data acquiring application on the electronic device100, such as the gallery application162, or (2) a VI component or a presentation layer hosted in the VI component.

As another example, assume the VI management system300receives, from a module260utilizing a model320for object recognition, a visual classifier result identifying an animal captured in visual data. As described in detail later herein, based on an ontology of models320, the VI management system300may run the same module260again, or a different module260, utilizing a different model320for additional information related to the animal identified, such as geographical areas where the animal is prevalent.

In one embodiment, a user (e.g., a user of the electronic device100) may select a module260to perform a particular vision processing task on visual data. In another embodiment, an application250is configured to select one or more modules260to activate to perform a particular vision processing task.

FIG. 5illustrates an example application screen500for the camera application161, in one or more embodiments. In one embodiment, the camera application161is an AR camera application providing a virtual assistant that makes full use of the VI system200. The virtual assistant has one or more installed AR operating modes that provide information about content of images/videos captured live via the camera140or stored in the gallery application162. Each installed AR operating mode is configured to perform a particular vision processing task and corresponds to a presentation layer255hosted in the camera application161running one or more modules260for the vision processing task. For example, the one or more installed AR operating modes may include, but are not limited to, one or more of the following: (1) a MagicLens operating mode corresponding to a first presentation layer255hosted in the camera application161running one or more modules260of the DL module family410for object recognition, (2) a Landmark operating mode corresponding to a second presentation layer255hosted in the camera application161running one or more modules260of the Cloud module family460for landmark/location recognition (i.e., identifying a landmark or location present in an image/video frame), (3) a Text operating mode corresponding to a third presentation layer255hosted in the camera application161running one or more modules260of the Symbol module family440for text recognition, (4) a Face operating mode corresponding to a fourth presentation layer255hosted in the camera application161running one or more modules260of the Face module family420for face recognition, and (5) one or more third party operating modes, wherein each third party operating mode is customized/provided by a third party and corresponds to a fifth presentation layer255hosted in the camera application161running one or more modules260of the Track module family450for object tracking one or more objects present in image/video frames.

In one embodiment, the MagicLens operating mode is configured to run a first set of models320(i.e., base models) and, based on visual classifier results returned by the first set of models320, activating and running one or more additional models320.

As shown inFIG. 5, in one embodiment, the camera application161presents a user with an application screen500comprising one or more selectable graphical user interface (GUI) elements corresponding to the one or more installed AR operating modes for user selection. For example, as shown inFIG. 5, the application screen500may include, but is not limited to, one or more of the following GUI elements: (1) a first GUI element501corresponding to the Magic Lens operating mode, (2) a second GUI element502corresponding to the Landmark operating mode502, (3) a third GUI element503corresponding to the Text operating mode503, (4) a fourth GUI element504corresponding to the Face operating mode, and (5) a fifth GUI element505corresponding to a particular third party operating mode. In one embodiment, the application screen500further comprises a selectable GUI element506that provides the user with an option of downloading and installing one or more additional AR operating modes in the camera application161.

In one embodiment, the camera application161together with the VI management system300functions as a real-world browser, extending the capabilities of the camera application beyond just capturing images and videos. The camera application161connects a user's digital world to the real world, allowing the user to search, explore and learn more about content and context of objects, scenes, people and experiences present in visual data directly within the camera application161.

In one embodiment, the ability to install and download additional AR operating modes enables an AR application ecosystem that allows partners and application developers to provide new and updated AR applications to users of the electronic device100.

In one embodiment, the camera application161together with the VI management system300provides a mobile eCommerce channel for transactions via a mobile payment and digital wallet service provided by a device vendor of the electronic device100and/or one or more third party mobile payment and digital wallet services.

In one embodiment, the camera application161together with the VI management system300functions as an intelligent vision memory service that remember what a user has seen or interacted with via the camera application161and/or the gallery application162, and recommends and retrieves content and context of objects, scenes, people and experiences present in visual data on commands of the user.

FIG. 6illustrates another example application screen510for the camera application161, in one or more embodiments. Assume when a user of the electronic device100is utilizing the camera application161, the VI management system300receives sensor-based contextual information comprising location data captured by a GPS of the electronic device160, wherein the location data indicates that the user is at a movie theatre. A presentation layer255hosted in the camera application161may run a module260for poster recognition to process a poster within a camera view of the camera140to identify a movie that the poster corresponds to. As shown inFIG. 6, in response to receiving a visual classifier result indicative of the movie identified from the module260, the camera application161may present an application screen510to the user, wherein the application screen510displays the visual classifier result (i.e., name of the movie identified) and provides the user with an option of invoking a third party application for purchasing movie tickets for the movie identified.

FIG. 7illustrates another example application screen520for the camera application161, in one or more embodiments. Assume a user captures via the camera application161or selects from the gallery application162an image of a plant. A presentation layer255hosted in the camera application161may run a module260for object recognition to identify a type of plant present in the image. As shown inFIG. 7, in response to receiving a visual classifier result indicative of the type of plant identified (e.g., a succulent plant) from the module260, the camera application161may present an application screen520to the user, wherein the application screen520displays the visual classifier result (i.e., the type of plant identified) and provides the user with additional information related to the image, such as, but not limited to, a location of a store where the user may buy the plant, a guide for taking care of the plant, a third party application the user may utilize for purchasing the plant online, and another third party application the user may utilize for providing coupons when purchasing the plant online or in-person.

FIG. 8illustrates another example application screen530for the camera application161, in one or more embodiments. Assume a user captures via the camera application161or selects from the gallery application162an image of a food item. A presentation layer255hosted in the camera application161may run a module260for object recognition to identify a type of food item present in the image. As shown inFIG. 8, in response to receiving a visual classifier result indicative of the type of food item identified (e.g., a cheeseburger) from the module260, the camera application161may present an application screen530to the user, wherein the application screen530displays the visual classifier result (i.e., the type of food item identified) and provides the user with additional information related to the image, such as, but not limited to, a nutritional information for the food item, online reviews of a restaurant serving the food item, and a third party application the user may utilize for logging the nutritional information into his/her food journal (e.g., a diet/fitness object tracking application).

FIG. 9illustrates another example application screen540for the camera application161, in one or more embodiments. In one embodiment, the camera application161allows a user to switch between different AR operating modes based on one or more user interactions with the application screen540. For example, as shown inFIG. 9, the user may browse/scroll through the different AR operating modes using hand gestures (e.g., swiping left or right through different GUI elements corresponding to the different AR operating modes, as illustrated by directional arrow F).

With the VI management system300, there can be a large number of models320present on the electronic device100. However, due to limited computation resources on the electronic device100, it may not be feasible for all models320present on the electronic device100to be active simultaneously during run-time. In one embodiment, the VI management system300provides a functionality for loading/activating or unloading/deactivating one or more models320during run-time, thereby providing increased scalability to accommodate large numbers of models320that in turn improve recognition capabilities. In one embodiment, the VI management system300accommodates running multiple models320simultaneously in parallel. In one embodiment, the VI management system300accommodates running models320on-device (i.e., on the electronic device100) and on-cloud (i.e., remotely on a server200, such as a cloud service).

In one embodiment, an application may be deployed on an electronic device in a two-stage process: 1) a learning process in which one or more pre-trained models are created using complex neural network architectures with large amounts of training data, and 2) an inference process in which inference is performed based on a pre-trained model. Specifically, in the inference process, once one or more pre-trained models are available, an application may be developed to use one or more recognition capabilities available from the one or more pre-trained models for analyzing visual data. This may be used to develop one or more AR applications for recognizing a particular set of objects using the one or more pre-trained models. Further, each application is tied to a specific model, and each model may recognize a certain number of object categories based on the training data.

In one embodiment, the VI management system300can leverage benefits of performing inference at run-time based on multiple models320, and also switch models320at run-time based on contextual information of a user of the electronic device100(“user context”). Examples of user context include, but are not limited to, GPS location, application usage, user intent, etc.

In one embodiment, the VI management system300organizes pre-trained models in an intelligent and hierarchical manner based on categories and hierarchical relationships between the categories, wherein each hierarchical relationship (i.e., ontology) may be automatically generated (i.e., automatically derived), pre-defined (i.e., pre-determined), or provided by an application developer/user. A hierarchical tree structure enables recognition of generic/general categories followed by recognition of specific categories triggered by visual classifier results indicative of the generic/general categories. In one embodiment, the VI management system300supports adding to, and removing from, at run-time one or more categories with simple modifications/updates to a hierarchical tree structure, thereby providing a scalable and dynamic system architecture with a varying number of categories.

In one embodiment, the VI management system300may execute multiple models and/or multiple DL engines either in parallel or in sequence on the same visual data.

In one embodiment, the VI management system300allows for different models to be invoked/loaded in the electronic device100based on user context. The VI management system300may process visual data based on multiple models to gain varying levels of understanding about content and context of objects, scenes, people and experiences present in the visual data. In one embodiment, the VI management system300provides flexibility of running specific models using CPU resources of the electronic device100and specific models using specialized hardware of the electronic device100, such as GPU. In one embodiment, the VI management system300provides a hybrid approach for recognizing some object categories available on the electronic device100and some object categories from the cloud.

A domain is a set of models320. In this specification, let the term “model graph” generally refer to a directed graph representing a hierarchical tree structure of categories (e.g., object categories). In one embodiment, a model graph comprises multiple category nodes including: (1) a first category node representing a domain root indicative of a domain represented by the model graph, and (2) multiple additional category nodes representing multiple models320included in the domain. The model graph further comprises one or more hierarchical links between the multiple category nodes. Each hierarchical link represents a hierarchical relationship between a pair of models. In one embodiment, the model graph may include at least one of the following types of hierarchical links—category ontology links or category similarity links. As described in detail later herein with reference toFIG. 12, a category ontology link between a pair of models is based on ontology information relating to the pair of models. As described in detail later herein with reference toFIG. 14, a category similarity link between a pair of models is based on a similarity metric between the pair of models.

Examples of different types of domains include, but are not limited to, the following: (1) an Everyday domain representing everyday objects and animals recognizable by one or more models320, (2) a Places domain representing indoor and outdoor locations/landmarks recognizable by one or more models320, (3) a Cars domain representing car models recognizable by one or more models320, (4) a Food domain representing food items recognizable by one or more models320, (5) a Posters domain representing movie/film posters recognizable by one or more models320, (6) a Sign domain representing American Sign Language recognizable by one or more models320, (7) an Age domain representing age groups recognizable by one or more models320, (8) a Gender domain representing gender groups recognizable by one or more models320, (9) a Logos domain representing brand logos recognizable by one or more models320, (10) an Emotions domain representing basic user emotions recognizable by one or more models320, and (11) a Flowers domain representing flower types recognizable by one or more models320.

FIG. 10illustrates an example execution manager700, in one or more embodiments. In one embodiment, the VI management system300comprises an execution manager700for performing inference at run-time based on multiple models320, and switching models320at run-time based on contextual information (i.e., real-time management of models320during run-time). The execution manager700facilitates parallel model execution in which multiple models320are run simultaneously.

As described in detail later herein, in one embodiment, to determine which models320to execute during run-time based on context (e.g., user context) and application preferences (e.g., quality of service preferences), the execution manager700is configured to perform each of the following: (1) a domain selection process in which a domain suitable for visual data received by the execution manager700is selected, and (2) a model selection process in which one or more models320included in the selected domain are selected for activation during run-time.

In one embodiment, the execution manager700is configured to access, during run-time, a set740of active models (i.e., activated/loaded models)320maintained by the VI management system300. An active model320is a model320that is loaded into a corresponding active engine340that is running the active model320during run-time. An active engine340is an engine340that is active during run-time. The set740may comprise at least one of a community model321, a vendor model322, a user model323, and a third party/partner model324.

In one embodiment, the execution manager700comprises a model execution system710comprising a distribution unit711configured to: (1) receive, as input, visual data comprising one or more image/video frames (e.g., from the camera application161or the gallery application162), and (2) forward/pass the visual data to all active engines340running the set740of active models320. Each active engine340processes the visual data based on at least an active model320loaded into the active engine340, and generates a corresponding visual classifier result indicative of one or more classification labels (e.g., object categories) for one or more objects present in the visual data and/or one or more regions in the visual data where the one or more objects are detected. Each visual classifier result generated may have one or more associated confidence values, wherein each confidence value is indicative of an accuracy of the visual classifier result.

In one embodiment, all the active engines340run the active models320loaded into the active engines340simultaneously, thereby processing the visual data in parallel (i.e., parallel model execution). In another embodiment, the active engines340process the visual data sequentially. In yet another embodiment, the active engines340process the visual data partially in parallel and partially sequentially.

For example, assume the execution manger700is configured to run no more than N active engines340in parallel. Further, assume at some point during run-time, M models are required to process visual data, wherein N<M. The execution manager700may run the first N models in parallel using N active engines340, followed by groups of up to N models in parallel until all M models have been run.

In one embodiment, the model execution system710comprises an aggregation unit712configured to merge/combine all visual classifier results from all the active engines340(i.e., result aggregation).

In one embodiment, the model execution system710comprises a selection unit713configured to: (1) monitor each visual classifier result generated by each active engine340to determine when execution is completed, (2) remove duplicate visual classifier results, and (3) remove visual classifier results having associated confidence values that do not meet one or more pre-determined confidence thresholds (i.e., low confidence values).

In one embodiment, the execution manager700comprises a domain selection system720configured to automatically select a domain suitable for visual data received by the execution manager700(i.e., domain selection process), wherein the selected domain includes one or more models320that are candidates for activation during run-time. The domain selection system720is configured to automatically select a domain in accordance with, but not limited to, one of the following methods: hierarchy-based domain selection method, a feature vector-based domain selection method, or a temporal-based domain selection method.

In one embodiment, the domain selection system720comprises a hierarchy-based domain selector721configured to apply the hierarchy-based domain selection method. The hierarchy-based domain selection method comprises selecting a domain based on one or more hierarchical relationships (i.e., category ontology links and/or category similarity links) between models320included in a hierarchical tree structure. For example, assume a first domain is currently selected for processing the visual data. As an object present in the visual data is recognized by a model320included in the first domain, the hierarchy-based domain selector721may select a second domain suitable for further processing of the visual data by following a category ontology link or a category similarity link from the first domain to the second domain.

In another embodiment, the domain selection system720comprises a feature vector-based domain selector722configured to apply the feature vector-based domain selection method. The feature vector-based domain selection method comprises selecting a domain based on one or more feature representations (i.e., feature vectors) extracted from visual data. In one embodiment, for each available domain, the feature vector-based domain selector722is configured to determine a corresponding feature representation for the domain and store the feature representation with corresponding metadata information for the domain (e.g., in one or more storage units120). In response to receiving visual data, the feature vector-based domain selector722is configured to, for each image/video frame of the visual data, perform the following: (1) determine a corresponding feature representation for the image/video frame, (2) for each available domain, determine a corresponding distance metric representing a distance between the corresponding feature representation for the image/video frame and a stored feature representation for the domain, and (3) select a domain from the available domains having the smallest distance metric (i.e., having a feature representation that is closest to the corresponding feature representation for the image/video frame).

In another embodiment, the domain selection system720comprises a temporal-based domain selector723configured to apply the temporal-based domain selection method. The temporal-based domain selection method comprises selecting a domain based on one or more visual classifier results for one or more temporal windows. In one embodiment, for each available domain, the temporal-based domain selector723is configured to determine a corresponding feature representation for the domain and store the feature representation with corresponding metadata information for the domain (e.g., in one or more storage units120). In response to receiving visual data, the temporal-based domain selector723is configured to segment the visual data into multiple temporal windows, wherein each temporal window comprises a sequence (i.e., subset) of image/video frames included in the visual data. In one embodiment, the visual data may be segmented into multiple temporal windows based on scene boundaries. For each temporal window, the temporal-based domain selector723is configured to perform the following: (1) determine a corresponding feature representation for the temporal window, (2) for each available domain, determine a corresponding distance metric representing a distance between the corresponding feature representation for the temporal window and a stored feature representation for the domain, and (3) select a domain from the available domains having the smallest distance metric (i.e., having a feature representation that is closest to the corresponding feature representation for the temporal window).

In one embodiment, the execution manager700comprises a model selection system730configured to: (1) receive a selected domain from the domain selection system720, (2) determine, based on the selected domain, which models320to load/activate and unload/deactivate (i.e., model selection) during run-time (i.e., model selection process), and (3) determine, based on the selected domain, which engines340to activate or terminate (i.e., engine load balancing) during run-time. The model selection system730enhances user convenience as it removes the need for an application developer/user to explicitly select models for activation.

In one embodiment, the model selection system730applies a quality of service (QoS) scoring method to select a model from the selected domain to load/activate. Let r generally denote a visual classifier result comprising a classification label n determined using a model m. Let link(m, n) generally denote a set of category ontology links and category similarity links based on the model m and the classification label n. Let m′ generally denote a set of models320to which link(m, n) points. Let M′ generally denote a set of models320formed by taking the union of m′ over a set of visual classifier results. In one embodiment, the model selection system730comprises a model load balancer unit731configured to: (1) identify each model320in the set M′ as a candidate for activation, (2) determine a corresponding execution order for each model320in the set M′, and (3) coordinate activation of each model320in the set M′ in accordance with a corresponding execution order.

In one embodiment, the model load balancer unit731is configured to determine a corresponding execution order for each model320in the set M′ based on the following information: (1) QoS preferences information732, and (2) model metadata information733. The QoS preferences information732may comprise, but is not limited to, one or more of the following: (1) a pre-determined threshold of engines340that can be run simultaneously on-device, (2) resource requirements of engines340and models320(e.g., memory and special hardware, such as the GPU), and (3) current activations of active models320currently loaded into the active engines340, where some models320may need to remain active while others may be taken “offline” (i.e., deactivated/unloaded). The model metadata information733may comprise, but is not limited to, one or more of the following: (1) model priorities for all models320included in the selected domain, where some models320, such as those near a domain root of a model graph representing the selected domain, may be kept activated between passes, and (2) model residencies for all models320included in the selected domain, where each model320either has a model residency requiring the model320to run on-device (i.e., on-device model) or a different model residency requiring the model320to run on-cloud (i.e., on-cloud model).

In one embodiment, the execution manager700seamlessly combines on-device models and on-cloud models. For each on-device model, the model load balancer unit731activates the on-device model by activating/starting an engine340with appropriate hyper-parameters and loading the on-device model into the active engine340. For each on-cloud model, the model load balancer unit731initializes a client endpoint needed to communicate with a cloud service that will run the on-cloud model. The model load balancer unit731may exchange messages with each engine340activated and/or each client endpoint initialized (e.g., exchange of activation requests and activation replies).

FIG. 11is a flowchart of an example process750for automatically selecting a model320for activation, in one or more embodiments. In process block751, receive a selected domain (e.g., from the domain selection system720). In process block752, obtain QoS preferences (e.g., QoS preferences information732). In process block753, obtain metadata for all models included in the selected domain (e.g., model metadata information733). In process block754, for each model included in the selected domain, determine a corresponding latency score, a corresponding accuracy score, a corresponding memory score, and a corresponding hardware score based on the QoS preferences and metadata for the model. In process block755, for each model included in the selected domain, determine a corresponding overall score based on a corresponding latency score, a corresponding accuracy score, a corresponding memory score, and a corresponding hardware score for the model. In process block756, select a model with a highest overall score across all models included in the selected domain for activation during run-time.

In one embodiment, process blocks751-756may be performed by one or more components of the model selection system730, such as, but not limited to, the model load balancer unit731.

FIG. 12illustrates an example model graph800including category ontology links, in one or more embodiments. The model graph800comprises a domain root805indicative of the Everyday domain represented by the model graph800. The model graph800comprises multiple category nodes representing multiple models320such as, but not limited to, an objects recognition model810for recognizing everyday objects, a scenes recognition model820for recognizing scenes, a car model830for recognizing car models, and a nature model840for recognizing nature.

As shown inFIG. 12, the objects recognition model810is linked to one or more additional models320, such as a shoes model811for recognizing shoe types/brands, a bags model812for recognizing bag types/brands, and a clothes model813for recognizing clothing types/brands.

As shown inFIG. 12, the nature model840is linked to one or more additional models320, such as an animals model841for recognizing animals, a trees model842for recognizing trees, and a mountains model843for recognizing mountains.

In one embodiment, the hierarchy-based domain selector721assumes each object recognized by a model320is associated with a category node of a model graph, and further assumes a model320is associated with a set of category nodes. In one embodiment, the hierarchy-based domain selector721generates/forms a category ontology link between a first model m1and a second model m2if an object recognized by the first model m1has the same category node as one of the category nodes of the second model m2. In one embodiment, the model graph800comprises one or more category ontology links. For example, as shown inFIG. 12, the model graph800comprises a category ontology link806A between the animals model841and each of the following on-device models320: a dogs model841A for recognizing dog types/breeds. As shown inFIG. 12, the model graph800further comprises a category ontology link806B between the bags model812and each of the following on-cloud models320: a first brand model812A for recognizing bags of a first type of brand (e.g., Coach®), a second brand model812B for recognizing bags of a second type of brand (e.g., Kors®), and a third brand model812C for recognizing bags of a third type of brand (e.g., Kate Spade®).

In one embodiment, the hierarchy-based domain selector721may use normalized word spaces (e.g., WordNet) to generate category ontology links from objects recognized by one model to objects recognized by one or more other models.

FIGS. 13A-13Cillustrates a sequence of operations performed by the execution manager700for automatically selecting a model320for activation, in one or more embodiments. Specifically,FIG. 13Ais a model graph850illustrating one or more models320initially activated by the execution manager700, in one or more embodiments. As shown inFIG. 13A, in an initial state (i.e., before the execution manager700receives visual data), the execution manager700activates only the following two models320: (1) an objects recognition model860for recognition of everyday objects, and (2) a scenes recognition model870for recognition of scenes. All other models320shown inFIG. 13A, such as a cats recognition model861, a dogs recognition model862, a movies recognition model871, an outdoors recognition model872, and a posters recognition model871A, are inactive.

FIG. 13Bis an updated model graph850illustrating one or more additional models320activated by the execution manager700in response to receiving visual data, in one or more embodiments. As shown inFIG. 13B, in response to receiving visual data comprising an image of a dog, the objects recognition model860processes the visual data and generates a corresponding visual classifier result comprising a classification label “dog” with an associated 0.988 confidence value. Assuming the 0.988 meets one or more pre-determined confidence thresholds, the execution manager700then activates the dogs recognition model862based on a category ontology link between the objects recognition model860and the dogs recognition model862. All other models320shown inFIG. 13A, such as the cats recognition model861, the movies recognition model871, the outdoors recognition model872, and the posters recognition model871A, remain inactive.

FIG. 13Cis an updated model graph850illustrating one or more additional models320activated by the execution manager700in response to a visual classifier result, in one or more embodiments. In response to the visual classifier result generated by the objects recognition model860, another activate model320processes the visual data. Specifically, as shown inFIG. 13C, the dogs recognition model862processes the visual data and generates a corresponding visual classifier result identifying a dog breed of the dog present in the visual data. For example, the visual classifier result may comprise a classification label “Appenzeller” with an associated 0.461 confidence value.

FIG. 14illustrates the example model graph800including category similarity links, in one or more embodiments. In one embodiment, the hierarchy-based domain selector721generates/forms a category similarity link856between a first model m1and a second model m2if an object recognized by the first model m1has a category node that is similar to a category node of the second model m2based on similarity metrics. In one embodiment, the model graph800further comprises one or more category similarity links856. For example, as shown inFIG. 14, the model graph800comprises a category similarity link856between the scenes recognition model820and each of the following models: a movies model821for recognizing movies/films (e.g., movie/film posters) as an object “movie theater” recognized by the scenes recognition model820is similar to an object “movie” recognized by the movies model821, and the animals model841as an object “zoo” recognized by the scenes recognition model820is similar to an object “animal” recognized by the animals model841.

In one embodiment, the hierarchy-based domain selector721may use word similarity metrics on normalized word spaces (e.g., Word2Vec) to generate category similarity links from objects recognized by one model320to similar objects recognized by one or more other models320.

FIG. 15illustrates an example feature vector-based domain selection method, in one or more embodiments. As stated above, the feature vector-based domain selector722selects a domain in accordance with a feature vector-based domain selection method. Let Imagen, generally denote an image/video frame included in visual data received by the execution manager700, wherein n is a positive integer. In one embodiment, the feature vector-based domain selector722is configured to: (1) for each image/video frame Imagen, extract a corresponding intermediate feature representation F(Imagen) utilizing a model725for feature extraction, such as a neural network architecture comprising multiple layers (e.g., Layer 1, Layer 2, . . . , Layer Z-1, Layer Z), and (2) generate a model specific intermediate representation (MSIR)727for the visual data by applying a linear transformation G[ ] (e.g., averaging) over each intermediate feature representation F(Imagen) extracted (i.e., G[F(Image1), F(Image2), . . . , F(Imagen)]. In one embodiment, the MSIR727is maintained in an application cache726of the VI management system300for later use by the feature vector-based domain selector722(e.g., comparing against other MSIRs generated).

In one embodiment, the feature vector-based domain selector722compares feature representations for the visual data against feature representations for available domains, and selects a domain from the available domains having the smallest distance metric.

In one embodiment, if a cloud service is used, the use of feature vectors improves privacy as the feature vectors are sent to a cloud service instead of actual image/video frames.

FIG. 16illustrates an example implementation of the temporal-based domain selector723, in one or more embodiments. As stated above, the temporal-based domain selector723selects a domain in accordance with a temporal-based domain selection method. Let TemporalWindowxgenerally denote a temporal window comprising a sequence (i.e., subset) of image/video frames included in visual data received by the execution manager700over time, wherein x is a positive integer. In one embodiment, the temporal-based domain selector723is configured to segment the visual data into multiple temporal windows. In one embodiment, the temporal-based domain selector723is configured to determine, for each temporal window TemporalWindowx, a corresponding feature representation. The temporal-based domain selector723compares feature representations for the temporal windows against feature representations for available domains, and selects a domain from the available domains having the smallest distance metric.

Let rTemporalWindowxgenerally denote a visual classifier result737for a temporal window. In one embodiment, the temporal-based domain selector723is configured to determine, for each temporal window TemporalWindowx, a corresponding visual classifier result rTemporalWindowx. In one embodiment, each visual classifier result rTemporalWindowxdetermined for each temporal window TemporalWindowxis maintained in an application cache736of the VI management system300for later use by the temporal-based domain selector723.

FIG. 17illustrates an example application screen900driven by the parallel model execution, in one or more embodiments. In one embodiment, the virtual assistant of the camera application161generates an application screen900for display to a user. As shown inFIG. 17, the application screen900is an example interface including a camera view901of the camera140and a result card (i.e., information card)910. The result card910includes a visual classifier result911for an object present in the camera view901. In one embodiment, if the object present in the camera view901comprises text, the visual classifier result911may include an optical character recognition (OCR) of the text.

As shown inFIG. 17, the application screen900comprises one or more selectable GUI elements such as, but not limited to, the following: (1) a first component912for invoking a translation of the visual classifier result911into a selected language (e.g., Korean), (2), a second component913for scrolling through multiple result cards910using hand gestures (e.g., swiping left or right, as illustrated by directional arrow A), (3) a third component914for speech capabilities (e.g., providing a machine-generated reading of the virtual classifier result911), (4) a fourth component915for receiving user feedback as input, and (5) a fifth component916for invoking one or more cloud services to perform further search for additional information related to the visual classifier result911.

FIGS. 18A-18Ceach illustrate another example application screen930for displaying multiple results cards910, in one or more embodiments. Specifically,FIG. 18Aillustrates the application screen930including one result card910, in one or more embodiments. In one embodiment, the virtual assistant of the camera application161generates an application screen930for display to a user. The application screen930is an example interface comprising a pull-up list931of result cards910. As shown inFIG. 18A, only one result card910is displayed in the pull-up list931. The user may scroll/browse through the pull-up list931by swiping the pull-up list931up (as illustrated by directional arrow B) to expand the pull-up list931or swiping the pull-up list931down to decrease/dismiss the pull-up list931shown.

FIG. 18Billustrates the application screen930including multiple result cards910, in one or more embodiments. As shown inFIG. 18B, in response to the user swiping up the pull-up list, the application screen930updates to display additional results cards910. The user may select any one of the result cards910displayed to expand the selected result card910and view additional details related to the selected result card910.

FIG. 18Cillustrates the application screen930including an expanded result card910, in one or more embodiments. As shown inFIG. 18C, the expanded result card910may include additional details and/or additional images related to a virtual classifier result included in the result card910.

FIGS. 19A-19Ceach illustrate another example application screen920for displaying a result card910that is paired with an object present in the camera view901of the camera140, in one or more embodiments. Specifically,FIG. 19Aillustrates the application screen920displaying a first result card910paired with a first object present in the camera view901, in one or more embodiments. In one embodiment, the virtual assistant of the camera application161generates an application screen920for display to a user. As shown inFIG. 19A, the application screen920is an example interface including a camera view901of the camera140, a first result card910for a first object (Object1) present in the camera view901, and a first bounding box921highlighting a first region in the camera view901where the first object is detected. The user may scroll/browse through multiple result cards910included in the application screen920using hand gestures (e.g., swiping left, as illustrated by directional arrow C).

FIG. 19Billustrates the application screen920displaying a second result card910paired with a second object present in the camera view901, in one or more embodiments. As shown inFIG. 19B, in response to the user swiping the first result card910, the application screen920updates to display a second result card910for a second object (Object2) present in the camera view901, and a second bounding box922highlighting a second region in the camera view901where the second object is detected. The user may scroll through other result cards910(e.g., returning back to the first result card910or viewing other result cards910) using hand gestures (e.g., swiping left or right, as illustrated by directional arrow D).

FIG. 19Cillustrates the application screen920displaying an expanded second result card910, in one or more embodiments. As shown inFIG. 19C, the user may expand the second result card910by swiping the second result card910up (as illustrated by directional arrow E) to expand the second result card910and view additional details related to the selected result card910. As shown inFIG. 19C, the expanded second result card910may include additional details and/or additional images related to a virtual classifier result included in the second result card910. The user may dismiss the expanded second result card910by swiping the expanded second result car910down.

In one embodiment, if there are multiple result cards910available for a user to scroll/browse through, the result cards910may be organized based on accuracy of virtual classifier results included in the result cards910(i.e., confidence values).

FIG. 20is a flowchart of an example process1000for processing visual data, in one or more embodiments. In process block1001, classify one or more objects present in an input comprising visual data by executing a first set of models associated with a domain on the input, where each model of the first set of models corresponds to an object category, and each model is trained to generate a visual classifier result relating to a corresponding object category in the input with an associated confidence value indicative of accuracy of the visual classifier result.

In process block1002, aggregate a first set of visual classifier results based on confidence value associated with each visual classifier result of each model of the first set of models.

In process block1003, select a second set of models to execute on the input based on the aggregated first set of visual classifier results and one or more hierarchical relationships between the first set of models and one or more other models.

In process block1004, determine an execution order of the second set of models based on one or more quality of service (QoS) preferences and model metadata information corresponding to the second set of models.

In process block1005, execute the second set of models in accordance with the execution order.

In one embodiment, process blocks1001-1005may be performed by one or more components of the VI management system300, such as the execution manager700.

FIG. 21is a high-level block diagram showing an information processing system comprising a computer system600useful for implementing the disclosed embodiments. Computer system600may be incorporated in an electronic device100or a server device (e.g., a server200). The computer system600includes one or more processors601, and can further include an electronic display device602(for displaying video, graphics, text, and other data), a main memory603(e.g., random access memory (RAM)), storage device604(e.g., hard disk drive), removable storage device605(e.g., removable storage drive, removable memory module, a magnetic tape drive, optical disk drive, computer readable medium having stored therein computer software and/or data), viewer interface device606(e.g., keyboard, touch screen, keypad, pointing device), a communication interface607(e.g., modem, a network interface (such as an Ethernet card), a communications port, or a PCMCIA slot and card), and a camera609. The communication interface607allows software and data to be transferred between the computer system and external devices. The system600further includes a communications infrastructure608(e.g., a communications bus, cross-over bar, or network) to which the aforementioned devices/modules601through607are connected.

Information transferred via communications interface607may be in the form of signals such as electronic, electromagnetic, optical, or other signals capable of being received by communications interface607, via a communication link that carries signals and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an radio frequency (RF) link, and/or other communication channels. Computer program instructions representing the block diagram and/or flowcharts herein may be loaded onto a computer, programmable data processing apparatus, or processing devices to cause a series of operations performed thereon to generate a computer implemented process. In one embodiment, processing instructions for process750(FIG. 11) and process1000(FIG. 20) may be stored as program instructions on the memory603, storage device604and the removable storage device605for execution by the processor601.

Embodiments have been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products. Each block of such illustrations/diagrams, or combinations thereof, can be implemented by computer program instructions. The computer program instructions when provided to a processor produce a machine, such that the instructions, which execute via the processor create means for implementing the functions/operations specified in the flowchart and/or block diagram. Each block in the flowchart/block diagrams may represent a hardware and/or software module or logic. In alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures, concurrently, etc.

The terms “computer program medium,” “computer usable medium,” “computer readable medium”, and “computer program product,” are used to generally refer to media such as main memory, secondary memory, removable storage drive, a hard disk installed in hard disk drive, and signals. These computer program products are means for providing software to the computer system. The computer readable medium allows the computer system to read data, instructions, messages or message packets, and other computer readable information from the computer readable medium. The computer readable medium, for example, may include non-volatile memory, such as a floppy disk, ROM, flash memory, disk drive memory, a CD-ROM, and other permanent storage. It is useful, for example, for transporting information, such as data and computer instructions, between computer systems. Computer program instructions may be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the embodiments has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention.