AUTOMATED FEEDING SYSTEM FOR FISH

Methods, systems, and apparatus, including computer programs encoded on computer-storage media, for the automated feeding of fish. In some implementations, a corresponding method may include obtaining meal configuration data including one or more parameters indicating a meal plan for feeding farmed fish; executing the meal plan based on the meal configuration data; receiving sensor data from one or more sensors during execution of the meal plan; and adjusting the execution of the meal plan based on the sensor data from the one or more sensors.

FIELD

This specification generally describes enhanced feeding systems for fish in aquaculture environments.

BACKGROUND

Researchers and fish farm operators face several challenges when feeding fish. Providing more food than what is required for normal growth may result in food waste as well as a corresponding increase in food expenses. Alternatively, providing less food than what is required for normal growth may affect the health of farmed fish, and may reduce the quality of the final product.

A manual process of observing and adjusting feed provided to fish is often used to monitor feeding levels, but this approach requires workers to be on site to monitor fish pen activity. Such a manual process is often time-consuming, expensive, and has several limitations, such as when cameras used to monitor fish activity are not correctly positioned or when adverse weather conditions decrease the availability of human observers.

SUMMARY

In general, innovative aspects of the subject matter described in this specification relate to the monitoring and imaging of fish and fish feed, for example in the context of aquaculture. In one example implementation, feeding sessions are defined by meal configuration data which specifies at least one value that is determined based on a sensed feeding condition. For example, a feeding rate, which is controlled by a blower of a feeding system, can be determined based on the depth that feed pellets are seen to be falling within the pen by a camera.

According to another example implementation of the subject matter, configuration data of a meal plan is used to control a feeding device for fish. Images of the fish obtained by a camera during feeding are analyzed to determine subsequent actions based on the configuration data. Researchers or fish farm operators may adjust parameters of the configuration data to adjust the feeding of fish. The parameter adjustments may be used to determine optimal feeding, including maximizing growth while minimizing food waste.

Advantageous implementations can include a submersible camera device that obtains images of fish and the surrounding environment. The submersible camera device may patrol a pen containing fish and capture images that may be processed by onboard computers or may be sent to a remote computer for processing. The submersible camera may capture images that may be used to determine, among other things, if fish are still feeding or if feed is falling below a threshold indicating that fish are likely done feeding.

Researchers or fish farm operators may adjust parameters of the configuration data iteratively in order to optimize the feeding process. By automating the feeding, unintentional changes to a specified meal plan caused by manual processes may be reduced. In this way, feeding may be more correlated with specific, configurable parameters of the configuration data enabling more accurate A/B testing as well as optimization.

One innovative aspect of the subject matter described in this specification is embodied in a method that includes obtaining meal configuration data including one or more parameters indicating a meal plan for feeding farmed fish; executing the meal plan based on the meal configuration data; receiving sensor data from one or more sensors during execution of the meal plan; and adjusting the execution of the meal plan based on the sensor data from the one or more sensors.

The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. For instance, in some implementations, executing the meal plan based on the meal configuration data includes: determining the one or more parameters of the meal configuration data; and sending a signal to a feeding mechanism indicating the one or more parameters of the meal configuration data.

In some implementations, the signal is configured to instruct the feeding mechanism how to feed the farmed fish based on the one or more parameters.

In some implementations, adjusting the execution of the meal plan based on the sensor data includes: processing the sensor data using one or more models to generate values associated with one or more determinations; comparing the values with one or more conditions indicated by the meal configuration data; and adjusting the execution of the meal plan based on the one or more conditions indicated by the meal configuration data being satisfied or not satisfied based on the one or more determinations.

In some implementations, the one or more models include a trained neural network model.

In some implementations, the meal configuration data includes a policy that prevents, based on the sensor data, feeding the fish more than a predetermined amount.

In some implementations, the predetermined amount is expressed as a percentage of biomass indicating the biomass of the farmed fish to be fed.

In some implementations, the biomass of the farmed fish is determined by one or more trained models configured to determine a biomass for each fish depicted in one or more visual images.

DETAILED DESCRIPTION

FIG.1is a diagram showing an example of a system100for the automated feeding of fish. The system100includes a control unit102configured to control a feeding mechanism106based on configuration data120and image data130collected by imaging device110. The feeding mechanism106provides feed114, such as fish pellets or other fish food, to fish112inside fish pen108.

The imaging device110may be equipped with one or more cameras configured to capture visual images underwater. The imaging device110may also be equipped with one or more sensors to provide data to the control unit102, such as a light sensor and a depth sensor. The imaging device110may be communicably connected to the control unit102in order to send the image data130to the control unit102.

The feeding mechanism106can include a pipe connecting the pen108to a central feeding station that provides the feed114to the pen108. In some implementations, a distributor located at the pen108may be used to more evenly distribute the feed114within the pen108. For example, the distributor may move around the surface of the pen108while dropping the feed114for the fish112. In some cases, a device may be used to propel the feed114. For example, a blower that blows air or water with the feed114can be used to disperse the feed114.

In stage A, the feeding mechanism106is not yet providing any feed. In stage B, the control unit102receives the configuration data120. The configuration data120specifies meal parameters that may include an intensity at which to feed, a rate to increment the intensity of feeding from not feeding to feeding, a required depth of food after a given period of time, a required fraction of fish that are to be feeding, and a delay between the current meal plan execution and a subsequent meal plan execution. Other parameters may be specified in the configuration data120depending on implementation.

In stage C, the control unit102sends a signal to the feeding mechanism106to begin a meal based on the configuration data120. The control unit102may send a signal to increment the feeding intensity of the feeding mechanism106based on an increment parameter specified in the configuration data120. The increment parameter may determine the rate at which the feed114is progressively provided by the feeding mechanism106to the fish pen108from an initial feeding intensity specified in the configuration data120up to a maximum feeding intensity or until a determined condition is satisfied, such as feed depth.

Each of the initial feeding intensity, maximum feeding intensity, and maximum feed depth may be specified in the configuration data120. The time between intensity increments may further be specified by the configuration data120. For example, the feeding intensity may increase by 10 grams per second every minute.

The imaging device110sends the image data130to the control unit102. In the example ofFIG.1, the image data130is a form of raw data and the control unit102processes the raw data in order to determine subsequent actions based on the meal plan in the configuration data120. By performing processing at the control unit102instead of the imaging device110, the system100may enable a lighter and cheaper imaging device110that need not be equipped with processing capabilities required for image analysis. Furthermore, the imaging device110may use energy that would have been required for image analysis to aid in other actions, such as movement, capturing images, sending images, among others.

In some implementations, the imaging device110may move around the pen108to obtain one or more images of the fish112and the surrounding environment. The imaging device110may be equipped with propellers, jets, or other propulsive means to travel within the water of the fish pen108. The imaging device110may be attached to the pen108with rope or cable. Lengths of rope or cable used to connect the imaging device110to the pen108may be elongated or shortened in order to move the imaging device110around the pen108. Instead of, or in addition to, a depth sensor, the length of rope may be monitored in order to determine the effective depth of the imaging device110.

The control unit102receives the image data130from the imaging device110. The image data130includes one or more events corresponding to the fish112. For example, the image data130may include an image of a fish. The fish may be one of one or more fish referred to herein as the fish112. The control unit102may analyze the image of the fish using one or more analysis techniques including machine-learning methods. The fish may be identified based on a trained detection model trained to identify fish. The fish may further be identified as currently eating or swimming with mouth open using the same or a different detection model trained to identify the open mouth of a fish.

In the example ofFIG.1, the image data130includes an image of one or more pellets of the feed114. The control unit102may analyze the image of the feed114to identify the one or more pellets. The imaging device110may capture an image of the feed114dropping below a threshold specified in the configuration data120. After processing, the control unit102can determine, based on the detected feed114below the threshold, to send a stop signal to the feeding mechanism106.

The control unit102may determine the depth that the feed114drops to by processing the image data130. For example, the image data130may include a depth measurement sent by the imaging device110indicating the current depth of the imaging device110. The depth measurement may be determined using a depth sensor affixed to the imaging device110. The image data130may further include a tilt angle indicating what direction the imaging device110is pointed. The control unit102may use one or more elements of the image data130to determine the actual depth of objects detected within images received from the imaging device110. The control unit102may compare the actual depth of the objects to criteria specified in the configuration data120.

In stage D, the control unit102determines, based at least on the feed114falling through to a depth specified in the configuration data120, to send a stop signal to the feeding mechanism106. The stop signal is configured to stop the feed114being provided to the fish112of the fish pen108.

In some implementations, a depth range may be used to determine whether to increase or decrease feeding. For example, the configuration data120may specify a depth range of 5 meters (m) to 15 m. If the control unit102detects the feed114beyond 15 m, the control unit102may send a stop signal to the feeding mechanism106configured to stop providing the feed114to the fish112. Similarly, in some implementations, if the control unit102does not detect the feed114below 5 m, the control unit102may send an increase feeding signal to the feeding mechanism106configured to increase feeding.

In some implementations, the control unit102may decrease feeding based on the depth at which the feed114is detected. For example, in a range of 5 m to 15 m, the control unit102may detect the feed114at 11 m. Based on a predetermined scale that assigns regions of the range to percent feeding reductions, the control unit102can determine that the feed114at 11 m corresponds to reducing feed by 10 percent and can send a signal to the feeding mechanism106configured to reduce feeding by 10 percent.

The control unit102may later detect the feed114at 12 m and, based on the predetermined scale, send a signal to the feeding mechanism106to reduce feeding by 25 percent. The percent reductions may increase until a stop depth. If the feed114is detected beyond the stop depth, feed is reduced by 100 percent and the meal is over. In general, any depth range may be used and regions incorporated within may be associated with any feeding reductions specified in the configuration data120.

In some implementations, detecting the feed114may include detecting at least one pellet or item of food. For example, each item of the feed114may be detected to determine a depth of the feed. The deepest identified item may be used to determine the depth of the feed114. In some implementations, depths of one or more items of the feed114may be averaged in order to determine the depth of the feed114. For example, the control unit102may detect5pellets at depths of 5 m, 5 m, 5 m, 7 m, and 12 m. If averaging is used, the depth of the feed114based on the5detected pellets may be determined, by the control unit102, to be 6.8 m.

In some implementations, other metrics are used to determine the depth of the feed114. For example, each depth associated with each item of food detected may be compared with a reference depth, such as the maximum stop depth, to determine the error between the detected depths and the reference. The differences may be combined to produce a metric, such as a root-mean-square of the differences. The metric may be used to control the effect of outlier detections on the depth calculation for the feed114.

In some implementations, clustering algorithms may be used to determine the depth of the feed114. For example, the depth of each item of the feed114may be determined by the control unit102based on the image data130. A clustering algorithm, such as k-means clustering, may then be used to cluster a first and a second portion. The first portion may be used to determine the depth of the feed114and the second portion may be excluded from the calculation as outliers.

In some implementations, other outlier detection methods may be used to exclude outliers from a depth of the feed114calculation. For example, the depth of each item of the feed114may be determined by the control unit102based on the image data130. Using any number of outlier detection methods including z-score or extreme value analysis, probabilistic and statistical modeling, linear regression models, proximity based models, information theory models, and high dimensional outlier detection methods, the control unit102can determine one or more outliers to exclude in order to determine the depth of the feed114.

In some implementations, the control unit102detects one or more other events based on the image data130. For example, the control unit102may detect a fraction of total feeding. The fraction of total feeding may be determined based, in part, on determining the ratio of fish with open mouths to fish with closed mouths using one or more detection models as described herein. The fish with open mouths may be considered eating while the fish with closed mouths may be considered not eating. The control unit102may determine whether to increase feeding, decrease feeding, or stop feeding based, at least in part, on the fraction of total feeding and a goal fraction of total feeding specified in the configuration data120.

In some implementations, the control unit102performs other actions in response to detecting one or more events in the image data130. For example, the control unit102may determine that the feed114does not drop to a predetermined depth. This may be the result of the fish112near the surface of the pen108eating the food before it drops to the predetermined depth. The configuration data120may specify that the feed114drop to a predetermined depth in order to ensure enough food is provided to the fish112. The control unit102, using both the image data130and the configuration data120, can send a signal to the feeding mechanism106that is configured to increase a feeding intensity in order to satisfy a depth of food requirement specified in the configuration data120.

For another example, the control unit102may determine that the feed114drops below a predetermined depth. The configuration data120may specify that the feed114not drop to, or below, a predetermined depth. If the control unit102determines that the feed114satisfies a drop threshold, the control unit102can configure and send a signal to the feeding mechanism106to reduce the feed114or stop the feed114being provided to the fish112.

In some implementations, the control unit102determines one or more values corresponding to one or more parameters of the configuration data120and, in response, determines a subsequent action based on the one or more values. For example, the control unit102can determine that the feeding rate of the fish pen108is lower than a predetermined threshold and that the feed drop depth is also lower than a predetermined threshold. The control unit102may then, based on the determined feed rate and determined feed drop depth compared to corresponding parameters in the configuration data120, configure and send a signal to the feeding mechanism106to provide more feed.

In another example, the control unit102can determine that the feeding rate of the fish pen108is lower than a predetermined threshold and that the feed drop depth is higher than a predetermined threshold. This may be caused by the fish112being full and not eating and therefore the feed114dropping further. The control unit102may then, based on the determined feed rate and determined feed drop depth compared to corresponding parameters in the configuration data120, configure and send a signal to the feeding mechanism106to reduce the feed114or stop the meal. In this way, a single value associated with a parameter of the configuration data120may be interpreted by the control unit102according to one or more other values determined by the control unit102.

In some implementations, the imaging device110may pre-process images obtained by the imaging device110. For example, the imaging device110may process obtained images and send corresponding processed data to the control unit102. In this way, the imaging device110may reduce processing requirements of the control unit102, reduce necessary bandwidth to communicate with the control unit102, as well as reduce energy requirements for the control unit102.

In some implementations, one or more additional imaging devices may be used to obtain images. For example, the imaging device110may represent any number of imaging devices that may be employed within the pen108to capture images.

FIG.2is a flow diagram illustrating an example of a process200for the automated feeding of fish. The process200may be performed by one or more electronic systems, for example, the system100ofFIG.1.

The process200includes obtaining meal configuration data indicating a meal plan (202). For example, as shown in stage B ofFIG.1, the control unit102may receive the configuration data120. Table 1 shows an example implementation of parameters and associated values that may be included in the configuration data120.

In some implementations, the configuration data120may include one or more adjustable parameters. For example: the configuration data120may include an “initial_intensity_grams_per_sec” parameter that can control how much of the feed114is provided to the fish112at the beginning of a meal; the configuration data120may include an “increment_intensity_grams_per_sec” parameter that can control how much the rate at which the feed114is provided to the fish112is increased; the configuration data120may include a “time_between_intensity_increment_in_minutes” parameter that can control the rate at which the feed114is incremented by a determined increment value (e.g., Increment_intensity_grams_per_sec′); the configuration data120may include a “min_depth_in_m” parameter that can specify the minimum depth of the feed114or the minimum depth of a sensor, such as the imaging device110; the configuration data120may include a “fall_through_depth_in_m” parameter that can specify the depth that the feed114may reach as it falls in the water, as sensed by a camera; the configuration data120may include a “fall_thru_score_threshold” that can control the amount of the feed114that falls through and prevent excess fall through, as sensed by the camera; the configuration data120may include a “feed_adjustment_proportional_gain” that can control the rate at which the feed114is adjusted when adjustments are required to satisfy parameters of the configuration data120; the configuration data120may include a “fraction_of_total_feeding” that can specify the amount of fish of the fish112that should be feeding during a meal where a trained model may detect fish with mouths open as fish that are feeding and fish with mouths closed as fish that are not feeding; the configuration data120may include a “delay_to_next_meal_in_minutes” that can specify the amount of time between the current meal, such as the ‘Main Meal’ and a subsequent meal where the delay may be adjusted based on the amount of food provided to the fish.

The process200includes executing the meal plan (204). For example, the control unit102may send a signal to the feeding mechanism106to begin a meal based on the configuration data120. The configuration data120may include an “initial_intensity_grams_per_sec” parameter indicating at what initial rate the feed114should be provided to the fish112.

The process200includes receiving sensor data during execution of the meal plan (206). For example, the control unit102may use input from one or more sensors, such as the imaging device110, to determine values of one or more parameters and then compare the values determined based on the sensor data to the obtained parameters in the configuration data120.

The process200includes adjusting execution of the meal plan (208). For example, the control unit102can compare a value of “fall_through_depth_in_m” specified in the configuration data120to a determined feed depth. After determining that the feed has reached a depth deeper than the value of “fall_through_depth_in_m” specified in the configuration data120, the control unit102can send a signal to the feeding mechanism106to decrease or stop feeding.

FIG.3is a diagram showing an example of a system300for executing a meal. The system300includes a sensor302that communicates with a control unit305. The system300can be used to implement the techniques described herein. For example, the imaging device110could be an example of the sensor302and the control unit102could be an example of the control unit305.

In some implementations, the sensor302may be attached to, or a part of, a moving device. For example, the sensor302may be a camera attached to a submersible patrol device configured to move underwater. The sensor302may be a light sensor, proximity sensor, or other electronic apparatus configured to collect data. The sensor302may include computer hardware configured to obtain and send data.

The control unit305may send a first communication307to the sensor302. In some cases, the first communication307may include details of a patrol task. For example, the control unit305may send a signal instructing the sensor302to move to a specific area within a fish pen, such as the fish pen108. The control unit305may send a signal instructing the sensor302to obtain data from a particular region, such as images from a particular depth or of a location within a fish pen.

In some implementations, the control unit305may control a type of movement for the sensor302. For example, the control unit305may send a signal instructing the sensor302to move to a specific depth or may indicate allowable depths within which the sensor302may patrol. For another example, the control unit305may send a signal instructing the sensor302to move at a particular speed.

In some implementations, the control unit305may control what data is obtained by the sensor302. For example, the control unit305may send a signal instructing the sensor302to capture visual images at a specific frequency for a particular amount of time. This may be configured to coincide with other events, such as feeding events where more image data may be useful to have.

In response to the first communication307, the sensor302may send sensor data309to the control unit305. The sensor data309may include visual images obtained by the sensor302, such as the image data130. The sensor data309may include other data forms as well depending on the first communication307. For example, the sensor data309may include one or more determinations generated by the sensor302or a device communicably connected to the sensor302.

In some implementations, the sensor data309may include other forms of data. For example, the sensor data309may include feedback from various feeding mechanisms, such as the feeding mechanism106. The sensor data309may indicate the amount of feed currently being provided or other status information such as an operational status of a device to provide food to an area, such as the fish pen108.

The control unit305may process the sensor data309based on data pre-obtained by the control unit305. In some cases, the pre-obtained data may include the configuration data120. The pre-obtained data in the system300includes a model312, a strategy315, and a policy315. In some implementations, the configuration data120may include one or more elements of the model312, the strategy315, and the policy315.

The model312processes the sensor data309to make one or more determinations based on the sensor data309. For example, if the sensor data309includes an image of a fish, the model312may detect the occurrence of the fish within the sensor data309. The model312may be a collection of one or more trained models configured to detect particular occurrences in the feeding scenarios discussed herein. For example, the model312may be used to detect occurrences of fish with mouths open or a number of pellets within one or more images of the sensor data309.

In some implementations, the model312includes a biomass estimation. For example, one or more trained models of the model312may be configured to determine a biomass estimation for a fish based on one or more images of the fish. In some implementations, an untrained model is provided images of fish with associated biomasses. The untrained model may be trained by adjusting one or more of its parameters to generate output that matches the associated biomasses. In some cases, truss lengths between parts of a detected fish are used as input to a model for predicting biomass. For example, the processing techniques described in U.S. application Ser. No. 16/734,661, filed Jan. 6, 2020 and entitled “Fish Biomass, Shape, Size, or Health Determination”, which is incorporated herein by reference, may be used in processing by the control unit305.

The output of the model312may be used to interpret conditions of the strategy315as well as the policy317. For example, the model312may detect that a fish pen has been provided 3 percent of its collective biomass in feed. The strategy315may indicate, for example, based on feed drop depth, that feeding should continue. The policy317may include a max feed amount to stop feeding when necessary, such as when the total feed amount reaches a certain percentage, e.g., 3 percent, of a fish pen's biomass.

Even though the parameters of the strategy315may require additional feeding, if the max feed amount in the policy317is reached, the control unit305may send a signal to stop feeding. In this case, the policy317may be used to ensure that fish are not fed too much or too little. The policy317may include maximum and minimum feeding rates as well as maximum feeding amounts in weight and as a percentage of biomass.

Table 2 shows an example implementation of parameters and associated values that may be included in the policy317. In some implementations, the policy317may be included in the configuration data120.

In some implementations, the policy317may include one or more adjustable parameters. For example: the policy317may include a “min_pen_feeding_rate_kg_per_min” that may prevent a given feeding strategy from reducing a feeding rate below a predetermined value specified by the parameter; the policy317may include a “max_pen_feeding_rate_kg_per_m in” that may prevent a given feeding strategy from increasing a feeding rate above a predetermined value specified by the parameter; the policy317may include a “max_pen_feeding_amount_percent_biomass” that may prevent a given feeding strategy from continuing to feed a group of fish an amount of feed more than a percentage of biomass specified by the parameter; the policy317may include a “pen_total_feed_hard_limit_in_kg” that may prevent a given feeding strategy from continuing to feed a group of fish an amount of feed more than an amount of feed specified by the parameter.

In some implementations, parameters of the strategy312and the policy317may be adjusted to change the automated feeding process. For example, as discussed herein with respect to the configuration data120, elements of the control unit305including the model312, the strategy315, and the policy317may be pre-obtained by the control unit305in order to process the sensor data309. Output of the model312may be used to determine whether one or more conditions in the strategy315are met. These conditions are discussed herein with respect to the configuration data120. Before adjusting the meal, the control unit305may check one or more conditions of the policy317.

In some implementations, the conditions of the policy317may be given higher priority than conditions of the strategy315. For example, when, based on one or more determinations of the model312, a condition of the strategy315would increase feeding and a condition of the policy317would stop feeding, the control unit305may give the policy317high priority by stopping the feeding instead of increasing feeding.

The control unit305may determine, based on one or more outputs of the model312and conditions of the strategy315and the policy317a feed adjustment325. A feed adjustment signal320indicating the feed adjustment325may be sent to a relevant feeding mechanism configured to adjust the feed provided to fish, such as the fish within the fish pen108.

In some implementations, the control unit305may determine that a maximum feed amount has been reached. For example, regardless of conditions within the strategy315, a max feeding condition of the policy317may be satisfied. In this case, the control unit305may determine to stop feeding. The feed adjustment325may be configured to indicate that feeding should be stopped and a corresponding feed adjustment signal320may be sent to a feeding mechanism.

In some implementations, the control unit305may determine one or more conditions of the strategy315or the policy317without processing data in the model312. For example, an amount of feed provided to fish may be sent by the sensor302and compared with a condition in the policy317without further processing by the model312. The sensor302in this case may include a sensor fixed to a feeding mechanism, such as the feeding mechanism106, configured to measure the feed being provided to fish. As previously mentioned, although depicted as a singular sensor for illustration purposes, the sensor302may include more than one sensor configured to obtain data.

FIG.4is a diagram illustrating an example of a computing system used for the automated feeding of fish. The computing system includes computing device400and a mobile computing device450that can be used to implement the techniques described herein. For example, one or more components of the system100or the system300could be an example of the computing device400or the mobile computing device450, such as a computer system implementing the control unit102, the feeding mechanism106, the imaging device110, the control unit305, or the sensor302.

The computing device400is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device450is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, mobile embedded radio systems, radio diagnostic computing devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

The computing device400includes a processor402, a memory404, a storage device406, a high-speed interface408connecting to the memory404and multiple high-speed expansion ports410, and a low-speed interface412connecting to a low-speed expansion port414and the storage device406. Each of the processor402, the memory404, the storage device406, the high-speed interface408, the high-speed expansion ports410, and the low-speed interface412, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor402can process instructions for execution within the computing device400, including instructions stored in the memory404or on the storage device406to display graphical information for a GUI on an external input/output device, such as a display416coupled to the high-speed interface408. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. In addition, multiple computing devices may be connected, with each device providing portions of the operations (e.g., as a server bank, a group of blade servers, or a multi-processor system). In some implementations, the processor402is a single threaded processor. In some implementations, the processor402is a multi-threaded processor. In some implementations, the processor402is a quantum computer.

The memory404stores information within the computing device400. In some implementations, the memory404is a volatile memory unit or units. In some implementations, the memory404is a non-volatile memory unit or units. The memory404may also be another form of computer-readable medium, such as a magnetic or optical disk.

The storage device406is capable of providing mass storage for the computing device400. In some implementations, the storage device406may be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions can be stored in an information carrier. The instructions, when executed by one or more processing devices (for example, processor402), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices such as computer- or machine readable mediums (for example, the memory404, the storage device406, or memory on the processor402). The high-speed interface408manages bandwidth-intensive operations for the computing device400, while the low-speed interface412manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high speed interface408is coupled to the memory404, the display416(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports410, which may accept various expansion cards (not shown). In the implementation, the low-speed interface412is coupled to the storage device406and the low-speed expansion port414. The low-speed expansion port414, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device400may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server420, or multiple times in a group of such servers. In addition, it may be implemented in a personal computer such as a laptop computer422. It may also be implemented as part of a rack server system424. Alternatively, components from the computing device400may be combined with other components in a mobile device, such as a mobile computing device450. Each of such devices may include one or more of the computing device400and the mobile computing device450, and an entire system may be made up of multiple computing devices communicating with each other.

The mobile computing device450includes a processor452, a memory464, an input/output device such as a display454, a communication interface466, and a transceiver468, among other components. The mobile computing device450may also be provided with a storage device, such as a micro-drive or other device, to provide additional storage. Each of the processor452, the memory464, the display454, the communication interface466, and the transceiver468, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

The processor452can execute instructions within the mobile computing device450, including instructions stored in the memory464. The processor452may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor452may provide, for example, for coordination of the other components of the mobile computing device450, such as control of user interfaces, applications run by the mobile computing device450, and wireless communication by the mobile computing device450.

The processor452may communicate with a user through a control interface458and a display interface456coupled to the display454. The display454may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface456may include appropriate circuitry for driving the display454to present graphical and other information to a user. The control interface458may receive commands from a user and convert them for submission to the processor452. In addition, an external interface462may provide communication with the processor452, so as to enable near area communication of the mobile computing device450with other devices. The external interface462may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

The memory464stores information within the mobile computing device450. The memory464can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. An expansion memory474may also be provided and connected to the mobile computing device450through an expansion interface472, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory474may provide extra storage space for the mobile computing device450, or may also store applications or other information for the mobile computing device450. Specifically, the expansion memory474may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory474may be provide as a security module for the mobile computing device450, and may be programmed with instructions that permit secure use of the mobile computing device450. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory (nonvolatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier such that the instructions, when executed by one or more processing devices (for example, processor452), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory464, the expansion memory474, or memory on the processor452). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver468or the external interface462.

The mobile computing device450may communicate wirelessly through the communication interface466, which may include digital signal processing circuitry in some cases. The communication interface466may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), LTE, 5G/6G cellular, among others. Such communication may occur, for example, through the transceiver468using a radio frequency. In addition, short-range communication may occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown). In addition, a GPS (Global Positioning System) receiver module470may provide additional navigation- and location-related wireless data to the mobile computing device450, which may be used as appropriate by applications running on the mobile computing device450.

The mobile computing device450may also communicate audibly using an audio codec460, which may receive spoken information from a user and convert it to usable digital information. The audio codec460may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of the mobile computing device450. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, among others) and may also include sound generated by applications operating on the mobile computing device450.

CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

Particular embodiments of the invention have been described. Other embodiments are within the scope of the following claims. For example, the steps recited in the claims can be performed in a different order and still achieve desirable results.