Dynamic user interface mode selection based on physical activity detection

Techniques for improving the convenience of activating different computing applications on a mobile computing device are disclosed. Sensors associated with a mobile computing device (e.g., accelerometers, gyroscopes, light sensors, microphones, image capture sensors) may receive inputs of various physical conditions to which the mobile computing device is being subjected. Based on one or more of these inputs, the mobile computing device may automatically select a content presentation mode that is likely to improve the consumption of the content by the user. In other embodiments, image analysis may be used to access different mobile computing applications.

TECHNICAL FIELD

The present disclosure relates to selecting applications on mobile devices. In particular, the present disclosure relates to dynamic user interface mode selection based on activity detection.

BACKGROUND

Mobile computing devices are used throughout a variety of business environments to improve productivity. While often used as a tool to maintain a high rate of communication responsiveness (e.g., via a mobile phone application, a mobile email application, and the like), organizations are introducing mobile communication devices to improve user productivity for other types of tasks. Whether in manufacturing production environments or remote field work that includes multi-modal observation, the multiple input and output modes and the diversity of possible functions enable mobile computing devices to significantly improve worker productivity.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in a different embodiment. In some examples, well-known structures and devices are described with reference to a block diagram form in order to avoid unnecessarily obscuring the present invention.1. GENERAL OVERVIEW2. SYSTEM ARCHITECTURE3. SELECTING A CONTENT PRESENTATION MODE USING PHYSICAL MEASUREMENT SENSORS OF A MOBILE COMPUTING DEVICE3.1 SELECTING A CONTENT PRESENTATION MODE BASED ON WHETHER THE MOBILE COMPUTING DEVICE IS IN MOTION OR STATIONARY3.2 SELECTING A CONTENT PRESENTATION MODE BASED ON AN ORIENTATION OF THE MOBILE COMPUTING DEVICE4. ENGAGING AN APPLICATION BASED ON CONTENT OF A CAPTURED IMAGE5. EXAMPLE EMBODIMENTS6. COMPUTER NETWORKS AND CLOUD NETWORKS7. MISCELLANEOUS; EXTENSIONS8. HARDWARE OVERVIEW

1. GENERAL OVERVIEW

One or more embodiments improve the convenience of mobile computing devices and the convenience with which mobile computing devices may be used to engage multiple different computing applications. In some embodiments described below, sensors associated with a mobile computing device (e.g., accelerometers, gyroscopes, light sensors, microphones, image capture sensors) may receive inputs of various physical conditions around the mobile computing device.

Based on one or more of the inputs, the mobile computing device may detect an application executed by the mobile computing device and automatically select a content presentation mode that is likely to improve the consumption by the user of content associated with the executed application. In some examples, the system may detect whether a mobile computing device is in motion or not in motion. The system may present content for the executed application in a first content presentation mode when the system detects that the mobile communication device is in motion. The system may present content for the executed application in a second content presentation mode when the system detects that the mobile communication device is in not motion.

As an example, the mobile computing device may detect whether the mobile computing device is held by a user or is placed on a stable support surface. Based on this determination, the mobile computing device may use the techniques described below to select between different content presentation modes (e.g., visual or auditory). As another example, the mobile computing device detects a current orientation, whether vertical or horizontal, and selects between different content presentation modes based on the detected orientation. Different content presentation modes may also be selected based on a combination of detected inputs and an application that is presenting the content.

In some embodiments, the mobile computing device uses sensor input, include image sensor data, to not only select between content presentation modes but also to engage additional computing applications. In some examples, a trained machine learning model may analyze image data captured by a mobile computing device and engage particular computing applications upon detecting one or more features within the captured image.

2. ARCHITECTURAL OVERVIEW

FIG.1Aillustrates a system100in accordance with one or more embodiments. As illustrated inFIG.1A, system100includes a mobile computing device102, a machine learning (ML) application118, a data repository154and external resources158A,158B. In one or more embodiments, the system100may include more or fewer components than the components illustrated inFIG.1A. The components illustrated inFIG.1Amay be local to or remote from each other. The components illustrated inFIG.1Amay be implemented in software and/or hardware. Each component may be distributed over multiple applications and/or machines. Multiple components may be combined into one application and/or machine. For example, while some functions are shown inFIG.1Aas executed within the ML application118, it will be appreciated that in other embodiments these functions may be executed by the mobile computing device102. Operations described with respect to one component may instead be performed by another component.

Additional embodiments and/or examples relating to computer networks are described below in Section 6, titled “Computer Networks and Cloud Networks.”

In some examples, the mobile computing device102may be any type of mobile computing device (e.g., a mobile phone, a tablet computer, a laptop computer) that includes one or more sensors measuring various external conditions. The mobile computing device102may be capable of executing computing applications and/or communicating with another computing device capable of executing computing applications via a network. In some examples, the mobile computing device102may include one or more of a web browser, mobile application(s) (alternatively and equivalently referred to as “computing applications”), or other software application communicatively coupled to the mobile computing device102. In some examples, mobile applications executable by the mobile computing device102may include an image capture application for capturing still images and/or video images, a microphone for capturing verbal and/or sound stimuli, content presentation applications that operate in cooperation with a speaker and a visual display integrated with the mobile computing device. The mobile computing device102may also execute specific application, for example, calendar applications, communications applications (e.g., text, email, phone), as well as business applications, such as financial, inventory, expense reporting, or other business function applications.

The mobile computing device102shown inFIG.1Aincludes sensors106, user input/output devices110, and an application activity monitor114.

The sensors106of the mobile computing device102may include any sensors used to receive data regarding physical conditions surrounding or otherwise affecting with the mobile computing device102. Example sensors106include, but are not limited to, gyroscopes, accelerometers, temperature sensors (e.g., an infrared sensor), fingerprint sensors, light and/or image sensors, global positioning satellite (GPS) sensors, among others.

The user I/O device110of the mobile computing device102may include a microphone for capturing, recording, and/or storing sound stimuli, an image capture device for capturing, recording, and/or storing visual images (whether still images or video images). Also included in the user I/O devices110are a speaker for presenting audio content to the user and a screen (also referred to as a display) for presenting visual content to the user. Examples of visual text include text, still images, video, animated graphics, schematic plans, and combinations thereof.

The mobile computing device102also includes an application activity monitor114that may identify the various computing applications operating on the mobile computing device102. In some examples, the application activity monitor114may detect the execution of applications configured to operate sensors106and/or user I/O devices110. In other examples, the application activity monitor114may detect computing applications and/or processes currently executing on the mobile computing device. In other examples, the application activity monitor114may store a timeline and/or history of previously executed applications and/or processes.

Specific applications and other components of the mobile computing device102(e.g., memory, storage, processors, voltage regulators, display drivers, radio frequency (RF) transceivers) are not expressly shown for clarity but will be understood to be present. A mobile computing device102may interact with an embodiment of the machine learning application118that is instantiated as a cloud service using one or more communication protocols, such as HTTP and/or other communication protocols of the Internet Protocol (IP) suite.

The example machine learning (ML) application118illustrated inFIG.1Aincludes a mobile device sensor input122, orientation analysis logic124, a machine learning engine130, a frontend interface146, and an action interface150. In some embodiments, ML application118is a cloud service, such as a software-as-a-service (SaaS) or a web service. In other embodiments, the ML application118is operated as a dedicated system (e.g., in a private network, an “on-premises” computer system, a private distributed computer network system).

The mobile device sensor input122and the command generator142operate in configurations of embodiments of the system100in which analysis of mobile device sensor data and the commands resulting from the analysis are executed by the machine learning application118. In other embodiments, sensor data and the commands resulting from the analysis may, at least in part, be executed by the mobile computing device102itself.

The mobile device sensor input122receives sensor data transmitted from the mobile computing device102and executes an initial analysis in preparation for passing the received sensor data to the machine learning engine130. For example, the mobile device sensor input122may receive data of various types including, but not limited to, audio data, still image data, temperature data, accelerometer data, gyroscope data, light intensity data, and/or an identification of a mobile computing application being executed concurrently with sensor data collection. Upon receiving any one or more of these data types, the mobile device sensor input122may identify the sensor input type to the machine learning engine130and then pass the identified data to the machine learning engine130for analysis.

In some examples, one or more elements of the machine learning engine130may use a machine learning algorithm to identify or select a content presentation mode based on a set of sensor data. A machine learning algorithm is an algorithm that can be iterated to learn a target model f that best maps a set of input variables to an output variable, using a set of training data. A machine learning algorithm may include supervised components and/or unsupervised components. Various types of algorithms may be used, such as linear regression, logistic regression, linear discriminant analysis, classification and regression trees, naïve Bayes, k-nearest neighbors, learning vector quantization, support vector machine, bagging and random forest, boosting, back propagation, clustering, and/or neural networks.

In this example, the machine learning engine130includes training logic134, an image processor138, and a command generator142. However, as previously indicated, the components of system100, including the ML application104generally and/or the machine learning engine130specifically, may vary depending on the particular implementation.

In this example, the training logic134may receive a set of electronic documents as input (i.e., a training corpus). Examples of electronic documents include, but are not limited to, electronically stored images, electronically stored sensor data from a mobile computing device, instruction media output modes (e.g., auditory vs. visual) corresponding to the electronically stored images and/or sensor data, and the like. In some examples, images and/or sensor data used for training may be labeled with a corresponding media output mode.

In one example, the training logic134may execute ML training algorithms to build a trained ML model ultimately stored in the ML learning engine130. Once trained, the ML learning engine130may then be used in some embodiments according to the methods described below inFIGS.2and3. Example ML training algorithms and ML models include, but are not limited to, unsupervised and supervised learning algorithms (e.g., neural networks).

In some embodiments, the training logic134includes a feature extractor that is configured to identify feature values and generate feature vectors from training materials, such as those described above. The feature extractor may tokenize sensor input data in a training corpus and then generate feature vectors that include a sequence of values, with each value representing a different token.

In some examples, labels may be assigned to (or appended to) feature vectors and/or otherwise associated with constituent tokens of feature vectors. A feature extractor in the training logic134may append other features to the generated feature vectors. In one example, a feature vector may be represented as [f1, f2, f3, f4], where f1, f2, f3correspond to tokens and where f4is a feature represented by a token not necessarily extracted from the training corpus, such as a label. For example, a set of tokenized sensor data represented as a feature vector may be appended with a label that indicates a vertical orientation of a device, thereby associating that particular set of sensor data with a computing device orientation. In another example, another set of tokenized sensor data represented as a feature vector may be appended with a label that indicates a mode in which instructions are to be presented (e.g., audio or visual).

Using these training data, the ML engine130is configured to automatically learn, from the training logic134, which combinations of sensor data from the mobile computing device102are associated with particular content presentation modes.

The ML engine130is also configured to contemporaneously identify a content presentation mode upon receiving new (“target”) sensor data from a mobile computing device. In some types of machine learning models, target data may be subsequent analysis) may be executed upon determining a similarity (e.g., using cosine similarity) between a labeled training data and target data. In other types (such as neural networks), target data is analyzed by determining gradients for a cost function that is subsequently applied to a series of intermediate “layers” of calculation.

Some types of data received from the sensors106of the mobile computing device may be easily analyzed by the system. For example, gyroscope data from the sensors106of the mobile computing device102may include a set of numerical data that informs the machine learning engine130(via the mobile device sensor input122) of various coordinates and/or orientations in space of the mobile computing device102. Other types of data (e.g., some types of movement data, ambient temperature data, light intensity data, GPS coordinate data) may also be analyzed by the ML application118with a relatively low consumption of computational resources.

In some examples, these types of data that are efficiently processed may be analyzed to determine a content presentation mode using a rule-based approach. In the illustrated example, the orientation analysis logic124in the machine learning application118may store one or more rules that may, in some embodiments, be used to analyze sensor data and select a content presentation mode. In some examples, the use of rules in the orientation analysis logic124may be used instead of the trained machine learning analysis of the machine learning engine130. For example, the orientation analysis logic124may define a rule that associates a particular orientation (e.g., horizontal) with an audio presentation mode. The orientation analysis logic124may receive sensor data and, for data that is encompassed by one or more rules defined in the orientation analysis logic124, identify an orientation of the mobile computing device. The analysis generated by the orientation analysis logic124may then be used in cooperation with other elements of the machine learning application118to select a content presentation mode. Once selected, other elements of the machine learning application118may transmit an instruction to the mobile computing device to use the selected content presentation mode.

However, other types of data may involve an additional layer of processing by the machine learning engine130to interpret and/or process the data before using the data to select a content presentation mode. For example, image data captured by a camera of a mobile computing device102may require additional processing and/or analysis before the machine learning engine130may analyze the data as part of the process for determining a content presentation mode.

The machine learning engine130includes an image processor138that provides additional processing of image data. For example, the image processor138may generate image feature vectors of images transmitted from the mobile computing device102via the mobile device sensor input122. In some examples, the image processor138may generate image feature vectors using a trained “classifier” machine learning model (e.g., a Euclidean distance classifier), a trained neural network, and the like to interpret image data. These feature vectors may then be analyzed using a trained model to identify image features.

Once the image processor138has identified one or more features within image data, these features may be passed to other elements of the trained machine learning engine130. The trained machine learning engine130may use the processed image data to determine an orientation of the mobile computing device102and identify an appropriate content presentation mode using, at least in part, the image data.

For example, the feature vectors generated by the image processor138from image data may indicate image features that are associated with a transition from a vertical (or inclined) orientation of the mobile computing device102to a horizontal orientation of the mobile computing device102. Examples of image features associated with a vertical orientation of the mobile computing device102include walls, shelves, humans, vehicles, stacked objects, furniture, and the like. Examples of image features associated with a horizontal orientation of the mobile computing device102include ceilings, overhead lights, rafters, beams, overhead cranes, ventilation ducts, ceiling tiles, and the like. The transition from a vertical orientation to a horizontal orientation may be associated with, by the trained machine learning model, a transition from a visual content presentation mode to an auditory content presentation mode.

This example may also be instantiated without a machine learning model, but instead executed upon identification of the features using a vector analysis of images and a stored instruction set associated with one or more identified image features (e.g., a rule set).

In still other embodiments, the machine learning application118may use both the rule-based analysis of the orientation analysis logic124and the machine learning engine130to complement one another. For example, because of the variety of possible orientation (and other) data of the mobile computing device, the orientation analysis logic124and the machine learning engine130may both analyze data to identify an orientation. Some sensor data from the mobile computing device102may be easily and efficiently analyzed by the orientation analysis logic124whereas other, more ambiguous data may be analysis by the machine learning engine130. The cooperative analysis may be combined using confidence intervals (e.g., probability calculations), similarity scores, and the like to reach a conclusion regarding an orientation of the mobile computing device102.

The machine learning engine130may include other processing elements, not shown, for other types of specific data. Processing techniques similar to those described for the image processor138(e.g., classifiers, neural networks) may be applied to audio data or other data.

Upon identifying an orientation of the mobile computing device102and identifying a content presentation mode corresponding to the orientation, the command generator142may generate an instruction to be sent to the mobile computing device102. That is, the command generator142of the ML engine130may generate a command to be sent to the mobile computing device that instructs the mobile computing device102to present data in a particular content presentation mode based on the orientation as determined by the ML engine130.

Once the content presentation mode instruction is generated, the command generator142may interact with other elements of the ML application118to transmit the instruction to the mobile computing device102. In some examples, the command generator142communicates with one or both of the frontend interface146and/or the action interface150to communicate instructions to the mobile computing device102.

The frontend interface146manages interactions between ML application118and the mobile computing device102. For example, mobile computing device102may submit requests to perform various functions and view results through frontend interface146. In some embodiments, frontend interface146is a presentation tier in a multitier application. Frontend interface146may process requests received from the mobile computing device102, and translate results from other application tiers into a format that may be understood or processed by the clients. Frontend interface146may be configured to render user interface elements and receive input via user interface elements. For example, frontend interface146may generate webpages and/or other graphical user interface (GUI) objects. Client applications, such as web browsers, may access and render interactive displays in accordance with protocols of the internet protocol (IP) suite. Additionally or alternatively, frontend interface146may provide other types of user interfaces comprising hardware and/or software configured to facilitate communications between a user and the application.

The action interface150action interface150provides an interface for executing actions using computing resources, such as external resources158A,158B. Action interface150may include an API, CLI, or other interfaces for invoking functions to execute actions. One or more of these functions may be provided through cloud services or other applications, which may be external to ML application118.

In some embodiments, external resources158A,158B are network services that are external to ML application118. Example cloud services may include, but are not limited to, social media platforms, email services, short messaging services, enterprise management systems, verbal communication systems (e.g., internet based voice communications, text chat communications, POTS communications systems) and other cloud applications. Action interface150may serve as an API endpoint for invoking a cloud service. For example, action interface150may generate outbound requests that conform to protocols ingestible by external resources158A,158B. Action interface150may process and translate inbound requests to allow for further processing by other components of ML engine130. Action interface150may store, negotiate, and/or otherwise manage authentication information for accessing external resources158A,158B. Example authentication information may include, but is not limited to, digital certificates, cryptographic keys, usernames, and passwords. Action interface150may include authentication information in the requests to invoke functions provided through external resources158A,158B.

In one or more embodiments, the system100may include one or more data repositories154. A data repository is any type of storage unit and/or device (e.g., a file system, database, collection of tables, or any other storage mechanism) for storing data. Further, the data repository may include multiple different storage units and/or devices. The multiple different storage units and/or devices may or may not be of the same type or located at the same physical site.

A data repository, such as the data repository154shown, may be implemented or may execute on the same computing system as the machine learning application118. The data repository154may be communicatively coupled to the machine learning application118via a direct connection or via a network.

Information describing the operations of the machine learning application118may be implemented across any of components within the system100.

In one or more embodiments, the various elements of the system100refer to hardware and/or software configured to perform operations described herein. Examples of these operations for are described below with reference toFIGS.2,3, and4.

In one or more embodiments, interfaces146,150refer to hardware and/or software configured to facilitate communications between a user and the mobile computing device102or between the mobile computing device102and the machine learning application118. Interfaces146,150render user interface elements and receives input via user interface elements. Examples of interfaces include a graphical user interface (GUI), a command line interface (CLI), a haptic interface, and a voice command interface. Examples of user interface elements include checkboxes, radio buttons, dropdown lists, list boxes, buttons, toggles, text fields, date and time selectors, command lines, sliders, pages, and forms.

In an embodiment, different components of the interfaces146,150are specified in different languages. The behavior of user interface elements is specified in a dynamic programming language, such as JavaScript. The content of user interface elements is specified in a markup language, such as hypertext markup language (HTML) or XML User Interface Language (XUL). The layout of user interface elements is specified in a style sheet language, such as Cascading Style Sheets (CSS). Alternatively, interfaces146,150are specified in one or more other languages, such as Java, C, or C++.

For clarity of explanation,FIG.1Billustrates ranges of orientation associated with a horizontal orientation of a mobile computing device.FIG.1Bshows that a plane including the screen of the mobile computing device may be +/−10° above or below a plane parallel to a horizontal reference plane (e.g., a flat portion of the earth below the mobile computing device). In other examples, the mobile computing device may be considered to be in a “horizontal” orientation for angles between +/−20° or even as high as +/−40° relative to the parallel horizontal reference plane inFIG.1B.

FIG.1Billustrates locations of a microphone, a screen, and a sensor assembly of the mobile computing device. The sensor assembly, which may include one or more of gyroscopes, accelerometers, thermometers, and light sensors, is also shown for reference.

FIG.1Cillustrates ranges of orientation associated with a vertical orientation of a mobile computing device. In this example, the mobile device may be considered to be in a vertical or inclined orientation when a plane defined by a screen of the mobile computing device is inclined from +10° to +120° (i.e., “above”) relative to a plane parallel to the horizontal reference plane. In an analogous example, the mobile device may be considered to be in a declined (non-horizontal) orientation when the plane defined by the screen of the mobile computing device is declined from −10° to −120° (i.e., “below”) relative to a plane parallel to the horizontal reference plane. As with the scenario described inFIG.1B, the precise values of these angles may be adjusted in different examples.

3. SELECTING A CONTENT PRESENTATION MODE USING PHYSICAL MEASUREMENT SENSORS OF A MOBILE COMPUTING DEVICE

This Section 3, in the context ofFIGS.2,3, and4, describes example sets of operations selected upon detection of various conditions associated with a mobile computing device in accordance with one or more embodiments. One or more operations illustrated inFIGS.2,3, and4may be modified, rearranged, or omitted all together. Accordingly, the particular sequence of operations illustrated in these Figures should not be construed as limiting the scope of one or more embodiments.

More specifically, Section 3.1 (FIG.2) describes techniques in which a mobile computing device (or a system in communication with a mobile computing device) uses sensor data to detect whether a mobile computing device is in motion (e.g., held in a hand) or is stationary (e.g., placed on a support surface). Each of these conditions is associated with a particular content presentation mode (e.g., visual or auditory), which the system selects automatically.

Section 3.2 (FIG.3) describes techniques in which the mobile computing device (or an associated system) uses sensor data to detect an orientation of the mobile computing device. A particular content presentation mode is selected based on the orientation.FIG.4describes techniques in which image processing may be used to engage additional computing applications as a function of features detected in a captured image.

3.1 Selecting a Content Presentation Mode Based on Whether the Mobile Computing Device is in Motion or Stationary

As indicated above,FIG.2illustrates an example set of operations, collectively identified as a method200, in which a system selects between content presentation modes (also referred to equivalently as “user interface outputs” or “user interface output modes”) based on the detection of various physical conditions associated with a mobile computing device. These user interface outputs switch from one particular content presentation mode to a different content presentation mode based on the detected physical conditions. This improves the convenience with which a user may interact with a mobile computing device. Automatically switching between content presentation modes based on sensor data, without express instructions entered by a user, may also improve efficiency and/or productivity of a user by reducing interruptions to a task that the user is performing.

The method200may begin by the system detecting execution of an application by a mobile device and also detecting that the mobile device is in motion (operation204). Detecting execution of an application by the mobile computing device is accomplished using any number of techniques. For example, some elements of the system (e.g., the system100) may detect packets transmitted by the mobile device through, for example, a packet switched network. The system may detect an identifier or other characteristic information associated with the application being executed by the mobile computing device in header or payload portions of the detected packet(s). The detected identifier may be used to detect the execution of the application. For example, the packet may be addressed to a particular server or device in communication with the network that is associated with a particular application being executed (e.g., a dedicated application server). In other examples, the packet may include protocols, instructions, or data encoded for processing by a particular application. In other examples, the system may execute a deep packet inspection to identify the application executed by the mobile device.

The system may detect the motion of the mobile device using any of the techniques described above in the context ofFIG.1A. For example, the system may determine that the mobile device is in motion by analyzing gyroscope data as a function of time generated by a gyroscope integrated with the mobile device. In other examples, the system may detect movement of the mobile device by analyzing image data captured over a period of time by an image sensor used in a camera function of the mobile device. The system may identify one or more features within an image (e.g., a face, structure features such as windows, doors, any structure defining a line or shape used as a reference) and detect changes of location of the image feature within successive captured images by detecting changes in coordinates of particular pixel intensities used to form the image of the feature.

In one example, the system may even detect the subtle movements of the mobile device that are associated with the device being held by a user (e.g., in a hand) (operation206). For example, the operation of human muscles is associated with some degree of variability and/or instability in the exerted force. In some examples, an object (such as a mobile device) held by a user will vary around a location at a frequency or variability in location that are characteristic of the operation of human muscles. A device being held by a user while walking will exhibit a pattern of movement associated with a walking or running gait. The system may detect and identify these characteristic patterns in movement, even when subtle.

Because the operations of the method200are not necessarily a function of mobile device orientation, but rather on sensor inputs indication motion (whether vibration, rotation, translation) the mobile device may be oriented vertically or horizontally in any of the operations of the method200.

As described elsewhere herein, the system may use one or both of a rule-based approach to detecting movement of the mobile device and/or a trained machine learning model approach. As described above in the context ofFIG.1A, patterns in image data or sensor data may be associated with rules to assign movement to the device, or even a type of movement (e.g., held by a stationary user, held by a walking or driving user, attached to a support that is held by a user or attached to a moving vehicle). Similarly, a training corpus of sensor data (including still-image and video image data) may be used to associate certain patterns in the sensor data with movement and/or types of movement.

Upon detecting both execution of the application and motion associated with the mobile device, the system may then select a content presentation mode based upon the determination that the mobile device is in motion (operation208). The system may select the content presentation mode based on the rules and/or the trained machine learning model described above. For example, if the motion of the mobile device is associated with a user holding the mobile device, then the system may select a visual content presentation mode that uses an integrated display of the mobile device. The selection of this visual content presentation mode may be made based on the presumption (or training) that the user is holding the mobile device in a position that enables an integrated display to be viewed by the user.

In an alternative scenario, the system may select an auditory content presentation mode when detecting certain types of motion. For example, if the type of motion detected is associated with that of a moving vehicle, which will have a very different distribution of vibrations, speed and/or changes in location (as indicated by a GPS sensor in the mobile device), the system may select the auditory content presentation mode to avoid distracting the user with a visual display of content.

Based on the selection performed in the operation208, the system may present content in the selected mode (operation212). The system may access the content, encode the content according to the selected mode and then transmit the correspondingly encoded content for presentation. For example, upon determining that the content should be presented in an auditory content presentation mode in the operation208, the system may access a text-based content item and convert the text data to corresponding machine-generated voice data that is presented via a speaker in communication with the mobile device. Alternatively, upon determining that the content should be presented in a visual content presentation mode in the operation208, the system may access a text-based content item, and simply transmit it for presentation as text to be rendered on a display of the mobile device. Alternatively, an audio portion of audiovisual content may be presented on the mobile device, or a text explanation in an audiovisual content item may be converted to audio data and presented.

In some examples, the system may detect changes in the movement of the mobile device and alter the presentation mode accordingly. This change in content presentation mode is illustrated in the example operations216-228of the method200.

In this example, the system may detect that the mobile device that is executing the application is no longer in motion (operation216). The system may detect the execution of the operation and the cessation of motion using the techniques described above in the context of the operation204. In some examples, the cessation of motion may be associated with the mobile device being placed on a support surface (operation218). For example, the sensor data may indicate that the mobile device has been placed on a shelf, a desk, a floor, or even on a stationary support that is attached to a shelf, a desk, or a floor. The operation218may even include the mobile device being placed in a bracket that is attached to a stable anchor (e.g., a table, a wall, a support). In other examples, the cessation of motion may be associated with the mobile device being placed in a pocket of a stationary user.

In response to the operation216, the system may select a second content presentation mode for the executing application that is different from the first content presentation mode (operation220). For example, the system may select an auditory mode upon the device being stationary based on a presumption (or training of a machine learning model) that the visual display is no longer visible to the user. This may be the case for a mobile device placed horizontally on a surface, thereby rendering its display less visible to the user.

The content previously presented in the operation212in the first mode may then continue to be presented by the system in the second mode (operation228).

3.2 Selecting a Content Presentation Mode Based on an Orientation of the Mobile Computing Device

In an alternative embodiment, a system may use mobile computing device sensor data to detect a change in orientation of the mobile computing device. The detected change in orientation may be used to cause a change in content presentation mode.FIG.3illustrates an example set of operations, collectively identified as a method300, in which a system selects between content presentation modes based on a detected mobile device orientation. As with the method200, the method300may improve the convenience with which a user may interact with a mobile computing device. Automatically switching between content presentation modes based on sensor data, without express instructions entered through a user interface, may improve efficiency and/or productivity of a user by reducing interruptions to a task that the user is performing that may involve use of the mobile computing device.

The method300may begin upon the system receiving a first set of one or more sensor inputs from sensors associated with the mobile computing device. This first set of sensor inputs may correspond to a first orientation (operation304).

Examples of sensor data that may be used to determine an orientation of the mobile device include temperature data (operation308), accelerometer data (operation312), and/or gyroscope data (operation316). While not illustrated inFIG.3, it will be appreciated that GPS data, image data, or other types of data collected by sensors associated with a mobile device may be used to determine orientation of the device.

For example, some mobile devices are equipped with sensors, such as infra-red sensors, that may be used to detect temperature (operation308). In some cases, the temperature sensor may be configured to detect a temperature of an ambient environment in which the mobile device is disposed. In other examples, the temperature sensor may be a “spot” sensor that detects a temperature of a target or a portion of a target (e.g., 1 square centimeter at a distance of 1 meter) at which the sensor is directed. In some examples, a temperature sensor may optionally be integrated with an image sensor of a camera. In other examples the temperature sensor may be separate from, but in communication with, the mobile device.

The system may also use temperature data to determine orientation by, for example, detecting whether the mobile device is proximate to objects that are known to produce heat, such as ceiling lighting fixtures in an industrial, commercial, or warehouse setting. In other examples, temperature data may be used (in coordination with other sensor data) to determine whether the mobile device is proximate to a human user (e.g., by detecting body temperature). This proximity may be used to determine orientation.

The system may also use accelerometer data to determine an orientation of a mobile device (operation312). For example, using multi-axis accelerometer devices integrated with a mobile device (e.g., micro electro-mechanical system (MEMS) for x, y, and z axes), the system may track changes in orientation. In some examples, these data may be combined with gyroscope data (e.g., from a MEMS gyroscope) (operation316) to determine orientation. Alternatively, gyroscope data may itself be used for determining orientation without using (operation316). Data from a fingerprint sensor may also be used to determine orientation in some cases.

Some or all of these data may be collected over a period of time. For example, the system may track accelerometer data over a period of time and using these data identify a series of orientations including a current orientation.

As described above, particular ranges of these data and/or combinations of ranges from these different types of data may be associated with a particular orientation. The system may establish these associations by applying a set of one or more rules, or via a trained machine learning model. For example, receiving input via a fingerprint sensor may be associated with a vertical orientation. This orientation may be confirmed by application of a rule analyzing accelerometer and/or gyroscope data. Alternatively, a trained machine learning model may identify a mobile device orientation as vertical using these data in combination with its training.

Some or all of these data types may also be used to determine whether the device is in motion, as described above in the context of the method200.

Upon the system using the sensor input to identify the first orientation, the system may use the orientation to initiate a first mode for presenting content to a user (operation320). In some examples, as described above in the context of the method200, a visual display mode may be used to present content (operation324). In some examples, as described above in the context of the method200, an auditory display mode may be used to present content (operation328).

In some examples, the visual mode may be associated with a vertical orientation because the vertical orientation may be associated with the mobile device being held by the user or propped vertically to improve a viewing angle. This need not be the case and may vary depending on a training corpus and/or rule set adapted for a particular use. While the presumption that a vertical orientation improves visual display accessibility may be valid in a warehouse, healthcare, or industrial setting (where users are often upright and moving), this may not be the case in an academic setting (where users are often stationary and sitting). An analogous explanation is applicable to the selection of an auditory display mode. The choice of which content display mode to employ for a given orientation will vary on the training data set used to train a machine learning model, and therefore on the particular application.

The method300continues upon the system receiving a second sensor input from the mobile computing device (operation332). This second sensor input may correspond to a second orientation that is different from the first orientation. For example, the first orientation may be vertical and the second orientation horizontal, or vice versa. As described above, examples of sensor inputs include, but are not limited to, temperature data (operation336), accelerometer data (operation340), and gyroscope data (operation348).

Responsive to receiving this second sensor input from the mobile computing device, the system may change the content presentation mode from the first mode to a second mode (operation352). As described above in the context of the operation320, the presentation modes include a visual display mode (operation356) and an auditory display mode (operation360). In the operation352, the system may switch from a mode used in the operation320to a different mode in response to the detected change in orientation.

In one example, the system may transition from the visual display mode324to a second, different type of visual display mode358that presents less detail than the first visual display mode presented in the operation324. For example, the first visual display mode324may present text instructions, diagrams, and other content information that is useful when read or viewed in detail by the user. The reduced detail visual display mode358may present content that may be both visible and informative when viewed at a distance or at a glance (e.g., on the order of milliseconds instead of the seconds or tens of seconds (or longer) for the first visual display mode324). Examples of the reduced detail visual display mode358may include large font text (e.g., 16, 18, 24 point font vs. 8, 10, or 12 point font used in the first visual display mode324), color coded fields, enlarged images (or a magnified portion of an image that occupies a large proportion of a display) or the like. This reduced detail visual display mode358may thus improve the utility of visual images when a user is walking, driving, or is otherwise viewing the visual display with a brief glance.

4. ENGAGING AN APPLICATION BASED ON CONTENT OF A CAPTURED IMAGE

FIG.4illustrates a set of example operations, collectively referred to as a method400, that may activate, engage, or otherwise execute one or more applications on a mobile computing device in response to identifying features within an image captured by the mobile computing device.

The method400may begin by detecting activation of an image capture function on a mobile computing device (operation404). The system may detect this activation using an application or process monitoring system. In some examples, this monitoring system may be native to the mobile computing device. In other examples, the mobile computing device may be configured to transmit a signal to a remote system (e.g., a server) upon activation of the image capture function. The remote system may in turn detect activation of the image capture function. Native application monitoring systems and packet analysis techniques used for identifying activation of an application are described above.

The system may detect storage of an image that has been captured by the image capture function (operation408). For example, the system may identify a change to a file inventory and further identify an additional file with a file type extension associated with image data. In one embodiment of this example, the system may identify the addition of a locally stored file identified with a .PNG, .JPEG, .GIF file extension, all of which are associated with image data. In another embodiment of this example, the system may detect activation of the image capture function and transmission of the image data (e.g., encoded with protocols or data associated with one of the preceding file types or another file type) from an image capture application to a remote storage location. The system may also use file storage time stamps to detect a newly stored image files by comparing a storage time of a file with an image file extension stamp to a system clock time. In still another example, the system may detect a decrease in available non-volatile memory capacity that is associated with an indication of a captured image.

Regardless of how the captured image is detected, responsive to detecting the captured image, the system activates an audio recording function to capture an audio recording (operation412). The system may transmit an instruction (e.g., via an operating system application programming interface or analogous system) to the audio recording application to activate the audio recording application and present a user interface (or other indication of the activation) to the user.

The system may engage the audio recording function in the operation412using any of several different techniques. For example, a set of rules may be used to identify the occurrence of a set or sequence of events that, collectively, are used as a trigger by which the system activates the audio recording function (operation416). In one example, the system may detect (1) the storage of an image captured by the image capture function as described above and (2) one or more other applications that, collectively, are used to identify a rule that activates the audio recording function. An illustration of this example is presented below in the Example Embodiment section.

In another example, the system may include a trained machine learning model that recognizes image content and engages the audio recording function in response to features identified in the stored image (operation420). In this operation, the system may include a trained machine learning model (trained according to the techniques described above) that, in some embodiments, may use an image classifier to identify image features. Neural networks or other machine learning models may also be trained and used.

In the operation420, the trained machine learning model may be trained to recognize image features that are associated with triggering the audio recording function. In one illustration, also described below in more detail in the Example Embodiment section, a trained machine learning model may be trained to identify damaged product and activate an audio recording feature used to by a user to record information associated with a procedure for returning the damaged product.

More generally, the trained machine learning model may be trained to activate any application upon detecting features within an image. For example, continuing with the example above, the trained machine learning model may activate a return process application (or “return material authorization/RMA” application) upon identifying damaged product in a captured image.

The system may then store the audio recording captured by the audio recording function and further store an association between the audio recording and the stored image (operation428). For example, the system may store a common record identifier, a URL or other active “pointer,” or other logical database cross-reference linking the image data and the audio data. This reference linking the two stored data items improves the convenience of locating and accessing these related items.

As indicated above,FIGS.5A-5Cillustrate an example set of user interface outputs that the system selects based on the detection of various physical conditions associated with a mobile computing device. These user interface outputs switch from one particular content presentation mode to a different content presentation mode based on the detected physical conditions. This improves the convenience with which a user may interact with a mobile computing device. Automatically switching between content presentation modes based on sensor data, without express instructions entered through a user interface, may also improve efficiency and/or productivity of a user by reducing interruptions to a task that the user is performing.

FIG.5Aillustrates one example scenario in which a user is holding a mobile computing device504that is displaying visual instructions508for supplying a cart512with bins (shown inFIG.5C). The cart512is disposed on a floor514which, referring toFIG.1C, acts as a horizontal reference plane.

The mobile computing device may be any type of mobile computing device, including those described above. Furthermore, the mobile computing device504may include any of the sensors described above in the context ofFIG.1A, including but not limited to accelerometers, gyroscopes, lights sensors, cameras, fingerprint sensors, global positioning satellite (GPS) sensors, thermometers, and the like. These sensors are omitted fromFIGS.5A-5Cfor clarity of depiction. As also described above in the context ofFIG.1A, the mobile computing device504may include various user I/O devices, including a visual display506and a speaker510.

In this example, the mobile computing device504presents visual instructions508on the integrated visual display506. While text instructions are shown in this example, it will be appreciated that the mobile computing device504may use any type of visual communication to present instructions. Examples of other types of visual instructions508that may be presented on the display506include text, still images, video, animated graphics, schematic plans, and combinations thereof.

The sensors in the mobile computing device504detect sensor inputs. The system may use the detected sensor input to determine that the device504is being held by the user. Based on this detection, the mobile computing device504determines that presentation of visual instructions is appropriate.

For example, the mobile computing device504(or a system to which it is connected, such as ML application118) may use one or both of gyroscope and/or accelerometer data to determine the device504is being held using, for example, the techniques described in the context ofFIG.2. When held by a human user, the mobile computing device504may detect frequent, low amplitude, and highly variable accelerations and changes in orientation. The system may associate these variable movements with the characteristic operation of human muscles, which are not configured or capable of maintaining a fixed and unvarying position over time. For example, the mobile computing device504may detect accelerations that have a duration on the microsecond scale with amplitudes that vary from 0.5 mm to 1 cm over the course of 1 second or less, 2 seconds or less, or 3 seconds or less.

In another example, a variation of acceleration amplitudes and/or frequencies associated with a user holding the device504may be greater than 1 standard deviation, greater than 2 standard deviations, or greater than 3 standard deviations from a mean acceleration amplitude and/or mean frequency. In some cases, the variability associated with a user holding the device504will increase over time. This trend of increasing variability over time (e.g., over multi-second time periods), reflecting natural muscle fatigue, may also be associated with a user holding the device504by the method200.

These subtle and variable movements may be sampled periodically or continuously over characteristic time periods (e.g., 1 second, 3 seconds, 10 seconds) to distinguish from vibrations that may exist in the operating environment (e.g., a manufacturing facility) and that may have different characteristic amplitudes, frequencies, and variabilities from those characteristic of the operation of human muscles. For example, while a manufacturing environment may produce background vibrations (e.g., from the operation of machines, conveyor belts, air handling equipment, moving vehicles), these are more likely to have a different amplitude (e.g., a few microns or less), a different frequency, and less variation than those caused by a human holding the device504because of the repeatable nature of machine operations.

In another example, the system may identify a sequence of very different movements to detect whether a user is holding the device504. One illustration of an example sequence that may be associated with a user holding the device504is that of a first period of frequent, low amplitude, and highly variable accelerations and changes in orientation (as described above) that is followed by a gross movement that corresponds to a user placing the mobile computing device in a pocket or on a support surface. This second period that includes the gross movement may include a change in location on the order an arm length. Arm length distances may range from 0.5 meters to 1 meter. Arm movement velocities and/or accelerations may be associated with values on the order of 0.5 to 2 meters/second over the course of 1 second or less. A third period in this example sequence is that of a period of no acceleration and no change in orientation. This third period corresponds to the device504being placed on a support surface or in a pocket.

Regardless of the sensor measurements analyzed, upon determining that a user is holding the device504, the system may present content (in this case instructions) using a visually-based content presentation system. As shown inFIG.5A, the content is text content508presented on display506.

The system may progressively provide content (in this case step by step instructions) via a presentation mode selected based on whether the device is held by a user or is disposed on a support surface. Turning toFIG.5B, the mobile computing device504displays a combination of text content516and image content520to the user on the integrated display506. The image content presents an image of a combination of medium bins524and small bins528on cart512.

FIG.5Cillustrates an example condition in which the system automatically switches to a content presentation mode different from the content presentation mode illustrated inFIGS.5A and5Bwhen the device504was held by the user. As shown inFIG.5C, the device504has been placed on a support surface—in this case the cart512. Using the sensor data described above, the device504(or a system in communication with the device504, such as the ML application118) determines that the user is no longer holding the device504. Instead, the system determines that the device504is now oriented horizontally and/or is on a stable support surface. As described above, the system may determine the use of the support surface512to hold the device based on the change in orientation, and/or a reduction in position variability and/or orientation variability, a reduced frequency and/or amplitude of accelerations, reduced temperature (from being further away from a human), and the like. Responsive to this determination, the system switches to an auditory content presentation mode, thereby presenting a next instruction532using the speaker510(shown inFIG.5A) of the device504.

As shown inFIGS.5A and5B, the mobile device504is held in a vertical or vertically inclined orientation. This orientation is consistent with the illustration inFIG.1Cbecause the mobile device is oriented between +10° and +120° relative to the horizontal reference plane (i.e., the floor214). As described above, this orientation may be detected using one or more gyroscopes integrated within the mobile computing device504.

As also described above, the system presents visual content (e.g., text508,516, and image520) using the integrated display506of the mobile computing device504. The system selects a visual content presentation mode when the device504is vertically inclined based on the presumption that the device504is being held. The system selects this content presentation mode to improve visual access of a user.

FIG.5Cillustrates that the device504is disposed on a support surface512in a horizontal orientation relative to the reference plane. This orientation is consistent with the illustration presented inFIG.1B. Based on this, the device504switches its content presentation mode to an auditory mode, thereby using the speaker510to present auditory instructions532.

In other examples, the system may use additional sensor data to complement and/or confirm the orientation sensor data in preparation for switching between content presentation modes. For example, a temperature sensor (e.g., an infrared sensor directed toward a user when the device504is held in an inclined vertical orientation with the screen facing a user) may detect changes in local temperature that indicate that the mobile computing device504is no longer close to (i.e., held by) the user. When a temperature threshold (e.g., greater than an ambient air temperature of 70° F. or 74° F.) is exceeded, the system may select a visual display mode. Conversely, if the system detects a drop of the temperature below this threshold, the system may switch to an auditory presentation mode based on the presumption that the device504is further from the user and is less visible.

In another example, one or more of acceleration data and/or acceleration frequency (i.e., vibration) data may be used as described above in Section 3.1 to corroborate orientation data. In still another data, a fingerprint sensor may be used to detect disposition of the device504in a hand of the user and employ a visual content display mode. When a fingerprint is no longer detected, the system may switch to an auditory presentation mode.

Other systems described herein may engage additional computing applications, and their corresponding functions by detecting features in images captured by the mobile computing device.FIGS.6A and6Billustrate one embodiment of this technique.

FIG.6Aillustrates an example scenario in which a mobile computing device604is used to identify a damaged product602(a case of leaking cola cans) at an inventory gate (e.g., a receiving dock). The reporting of a damaged product unit generally will involve documentation so that the damaged unit may be returned to a supplier for credit. In this example, documentation may include a picture of the damaged product unit, an accompanying explanation or description that comments on the conditions of the product that led to the damage, a purchase order number, a shipment tracking number, a product identifier (e.g., a SKU), and other information that may be used for logistical and/or financial aspects of a return.

Collecting the discrete and different elements of information may be time consuming and laborious, even when using the mobile computing device. This may be because of the inconvenience of switching between different computing applications on the mobile computing device604to, for example, record an image of the damaged product602, record an audio file or a text file storing descriptive information related to the damaged product602, access an inventory management application to return the product, use a bar code scanner to record a product identifier, and store all of these discrete items in a common location associated with the return.

In light of these inconveniences, embodiments of the present disclosure may analyze content in a stored image and automatically activate one or more computing applications on the mobile computing device604to reduce the inconveniences described above. In some examples, the system may even store an identifier with each of the discretely stored information items that associates a recorded image with a recorded sound file. For example, the system may associate one or more of a bar code scanner result, an inventory management system transaction identifier, a return material authorization (RMA) identifier, and the like, with stored images.

For example, the mobile computing device604, with integrated display608, may capture in image612of the damaged product602using an image capture application606and a camera (not shown, disposed on a side of the device604opposite the display608).

The device604(or a system in communication with the device, such as ML application118) may analyze the captured image612and determine that additional, related information will be collected using a computing application different from the image capture application606. In this example, the device604determines that an audio recording application616is to be activated responsive to the analysis of the captured image612. Upon activating the audio recording application616, the user may record descriptive comments and observations of the damaged product602. While not shown, the system may also activate any of the other computing applications (e.g., bar code scanner, RMA application, image annotation application, voice command application, financial applications, inventory management systems) using the techniques described herein.

The system may also generate an association between the audio recording file and the captured image so that these two discrete content items, generated by and stored by different computing applications, may be linked together. In this case, the identifier620(“Damaged Good #: 1”) is associated with both the audio recording616and the captured image612.

In some examples, the system may activate the audio recording application (or other application) by analyzing the captured image612and identifying within the captured image612one or more features that act as triggers for the audio recording application. In one example, the analysis of the image is performed using a trained machine learning model to identify triggering features. In the example presented inFIGS.6A and6B, the system may be provided with a training set of images that include triggering features and are accompanied by data collected using another application. For example, the training set may include individually stored images of damaged goods, each of which is associated with an accompanying audio recording, text file, voice command activation file, RMA application file, or the like. The machine learning model may be trained to associate images of damaged goods with activation of one or more of these applications. With this training, the system may then analyze a newly captured image and, using the training, activate another application accordingly without any express user input. In some examples, the training may be provided in subsets so that particular product types and/or particular damage types may be associated with the activation of different applications. For example, the leaking cola case602that is identified in the captured image612may be associated with activation of the audio recording application616. In another example, a captured image of a torn unit of sale package (e.g., a blister pack configured to be placed on a checkout aisle hanging display) may be associated with activation of a supplier RMA application or a barcode scanning application.

For examples in which an audio file is captured, the system may convert the audio file to text and analyze the text using natural language processing techniques. This may enable storage of an accompanying transcript (identified using the common identifier620), semantic analysis of the file, as well as any other analysis.

In other examples, the system may activate the other application (in addition to the image capture application) based on a similarity analysis with reference images exceeding a similarity threshold. For example, reference images of damaged cola cases may be analyzed by image processing techniques and converted into feature vectors. Vector values associated with leaks and damaged cartons may be linked to activation of the other application (e.g., audio recording application616). When a new image (i.e., image612) is captures, this new image may be vectorized and the vector compared to the vectors of the reference images using a similarity analysis (e.g., cosine similarity). If the similarity analysis is above a threshold value (e.g., above 0.5), then the system may determine that the new image contains features similar to images for which the audio recording application was activated, and consequently activate the audio recording application.

FIG.6Billustrates an additional application activated in response to the analysis of the image612. The application624is a defect reporting application that is activated in responses to the system analyzing the image612and detecting the damage. In this case, the defect reporting application624allows a user to enter text-based defect characterizations (e.g., “No Bar Code”), as well as a defect severity score, among other information.

6. COMPUTER NETWORKS AND CLOUD NETWORKS

8. HARDWARE OVERVIEW

Computer system700further includes a read only memory (ROM)708or other static storage device coupled to bus702for storing static information and instructions for processor704. A storage device710, such as a magnetic disk or optical disk, is provided and coupled to bus702for storing information and instructions.