Patent Description:
<CIT> discloses systems and methods for recognizing objects. A method may be executable to receive a query from a robot. The query may include identification data associated with an object and contextual data associated with the object. The query may also include situational data. The method may also be executable to identify the object based at least in part on the data in the query received from the robot. Further, the method may be executable to send data associated with the identified object to the robot in response to the query.

In the document:<NPL>, a robot can recognize an object through vision processing based on a detected RFID code and downloaded visual descriptor information.

In the document: <NPL>, a robot performs a task based on task descriptions, object recognition models and a environment map which it has autonomously downloaded from the RoboEarth database.

The scope of protection is defined by the claims.

The present disclosure is generally directed to methods, apparatus, and computer- readable media (transitory and non-transitory) for downloading targeted object recognition modules that are selected from a library of candidate targeted object recognition modules (e.g., hosted in the cloud) based on various signals, such as a task to be performed by a robot. Each targeted object recognition module may facilitate inference of information about of an observed-but-not-yet-classified object in the robot's environment, such as an object type (e.g., "cup," "plate," "telephone," etc.) and/or pose (e.g., orientation, location, etc.). According to the claimed subject matter, an object recognition client is operated on the robot. In an example that is useful for understanding the claimed subject matter, an object recognition client may be operated on a computing system (e.g., a desktop computer, router, etc.) that is considered "local" to the robot (e.g., in the same building, at least partially controls robot operation, etc.). When the robot is assigned a task, the object recognition client downloads one or more targeted object recognition modules that are selected (e.g., by the object recognition client or by another component) from the library based on various signals, such as a task to be performed by the robot. That way, the object recognition client can dedicate limited computing resources to recognizing/classifying objects the robot is likely to encounter while performing the task, and resources are not wasted attempting to recognize/determine object types/poses of objects that are not likely to be encountered.

Suppose a robot is instructed to clear a kitchen table. When performing such a task, the robot is likely to encounter objects typically found in kitchens, such as dishware, cups, silverware, napkins, cutlery, and so forth. The robot is less likely to encounter non-kitchen-related objects such as power tools, cleaning supplies, electronics, etc. Accordingly, one or more targeted object recognition modules that facilitate inference of an object type and/or pose of kitchen-related objects are selected, e.g., by the object recognition client or by one or more cloud-based processes, from the library of candidate targeted object recognition modules. The selected modules are downloaded to the object recognition client. Based on the downloaded targeted object recognition modules and vision data obtained by one or more vision sensors, the object recognition client determines object types and/or poses of one or more objects on the kitchen table.

Targeted object recognition modules may take various forms. In some implementations, the targeted object recognition modules may take the form of object models (e.g., CAD-based) that the object recognition client uses to determine information about observed objects, such as object types and/or poses, using vision and/or depth data obtained by one or more vision sensors. In other implementations, the targeted object recognition modules may take the form of two-dimensional ("2D") patterns or profiles of objects that may be matched to portions of 2D image data (e.g., video frames) captured by one or more vision sensors. In yet other implementations, the targeted object recognition modules may include routines (e.g., state machines) that may be implemented/triggered by the object recognition client to infer object types and/or poses.

As noted above, the downloaded targeted object recognition modules are selected from the library locally, e.g., by the object recognition client, or they may be selected remotely, e.g., by what will be referred to herein as a "root object recognition server" operated by one or more processors forming part of the cloud. In the latter case, the object recognition client may provide, to the root object recognition server, various signals, such as attributes of a task to be performed by the robot, data indicative of one or more observed objects, data available resources of the robot, etc. The root object recognition server may then select one or more targeted object recognition modules based on the one or more signals.

Targeted object recognition modules are selected from the library based on attributes of a task to be performed by a robot and based on other signals. For example, in some implementations, targeted object recognition modules may be selected from the library based on resources available to the object recognition client. If the object recognition client is implemented on the robot, and the robot is low on battery power, then a relatively small number of targeted object recognition modules may be downloaded. In some implementations, one or more attributes of a data (e.g., wireless) connection between the object recognition client and the cloud may be considered--e.g., the less reliable and/or lower bandwidth of the connection, the fewer modules downloaded. According to the claimed embodiment of the invention, if an assigned task will take a robot to an area that has an unreliable and/or low-bandwidth data connection, targeted object recognition modules are downloaded to the robot before it departs, and/or the robot waits until targeted object recognition modules are downloaded before entering the area with the unreliable connection.

In some implementations, a task performance history of the robot may be considered when selecting targeted object recognition modules from the library. If the robot has historically only been operated to perform a select few types of tasks, then targeted object recognition modules associated with objects that have been historically encountered by the robot when performing those tasks may be favored. In other words, the object recognition client and/or root object recognition server may learn over time which objects are expected to be encountered while performing particular tasks, and may favor targeted object recognition modules associated with the expected objects when a robot is performing the same or similar tasks. In some implementations, vision data captured of the environment in which the robot operates may be considered. For example, captured vision data may be analyzed by the object recognition client to determine various "clues" about an observed object, such as its approximate size. Targeted object recognition modules that target objects of that size may then be favored.

In some implementations, the object recognition client may apply multiple downloaded targeted object recognition modules to a single unclassified object detected in vision data captured by one or more sensors. Each targeted object recognition module may provide a competing object classification. For example, one targeted object recognition module may provide a classification of "cup," and another targeted object recognition module may provide a classification of "bowl. " The object recognition client may select from the multiple competing classifications based on a variety of signals, such as a confidence measure associated with each classification, a time required to obtain each classification (longer processing times may indicate less certainty), a comparison of multiple classifications, and so forth.

A computer-implemented method is provided according to claim <NUM>.

In various implementations, the one or more targeted object modules may be selected from the library based at least in part on one or more expected objects or object types associated with the task assigned to the robot. In various implementations, the expected objects or object types may be determined based on objects encountered by one or more robots historically when performing tasks sharing one or more attributes with the task to be performed by the robot. In various implementations, the one or more targeted object recognition modules may be selected from the library based at least in part on available resources of the robot. In various implementations, the available resources of the robot may include one or more attributes of a wireless signal available to the robot. In various implementations, the one or more targeted object recognition modules may be selected from the library based at least in part on the vision data.

In various implementations, the one or more targeted object recognition modules may be selected from the library by the object recognition client. In various implementations, the one or more targeted object recognition modules may be selected from the library by the remote computing system based on one or more attributes of the task provided to the remote computing system by the object recognition client.

In various implementations, the one or more vision sensors may include a camera configured to obtain depth data, and the vision data comprises a point cloud associated with the environment. In various implementations, the one or more vision sensors may be integral with the robot.

In various implementations, the method may further include selecting, by the object recognition client, from multiple competing inferences of object types or poses provided by multiple downloaded targeted object recognition modules, a given inference of an object type or pose based on a measure of confidence associated with the given inference. According to the claimed subject matter, the one or more processors are integral with the robot. In various examples useful for understanding the claimed subject matter, the one or more processors may be integral with a computing device that is in wireless communication with the robot.

In another aspect, a computer-implemented method may include the following operations: receiving, by a root object recognition server, from a remotely-operated object recognition client, data indicative of one or more observed objects, wherein the data indicative of one or more observed objects is based on vision data capturing at least a portion of an environment in which a robot operates; selecting, by the root object recognition server, one or more targeted object recognition modules from a library of targeted object recognition modules based at least in part on the data indicative of one or more observed objects, wherein each targeted object recognition module facilitates inference of an object type or pose of an observed object; and downloading, by the root object recognition server to the object recognition client, the selected one or more targeted object recognition modules.

According to claim <NUM>, a non-transitory computer readable storage medium is provided. Yet another implementation may include a control system including memory and one or more processors operable to execute instructions, stored in the memory, to implement one or more modules or engines that, alone or collectively, perform a method such as one or more of the methods described above.

<FIG> is a schematic diagram of an example environment in which selected aspects of the present disclosure may be practiced in accordance with various implementations. A robot <NUM> may be locally controlled in whole or in part by a control system <NUM>. Robot <NUM> may take various forms, including but not limited to a telepresence robot (e.g., which may be as simple as a wheeled vehicle equipped with a display and a camera), a robot arm, a humanoid, an animal, an insect, an aquatic creature, a wheeled device, a submersible vehicle, an unmanned aerial vehicle ("UAV"), and so forth. In various implementations, robot <NUM> may include logic <NUM>. Logic <NUM> may take various forms, such as a real time controller, one or more processors, one or more field-programmable gate arrays ("FPGA"), one or more application-specific integrated circuits ("ASIC"), and so forth. In some implementations, logic <NUM> may be operably coupled with memory <NUM>. Memory <NUM> may take various forms, such as random access memory ("RAM"), dynamic RAM ("DRAM"), read-only memory ("ROM"), Magnetoresistive RAM ("MRAM"), resistive RAM ("RRAM"), NAND flash memory, and so forth.

In some implementations, logic <NUM> may be operably coupled with one or more operational components <NUM><NUM>-n, one or more end effectors <NUM>, and/or one or more sensors <NUM><NUM>-m, e.g., via one or more buses <NUM>. As used herein, an "operational component" <NUM> of a robot may broadly refer to actuators, motors (e.g., servo motors), joints, shafts, gear trains, pumps (e.g., air or liquid), pistons, drives, or other components that may create and/or undergo propulsion, rotation, and/or motion. Some operational components <NUM> may be independently controllable, although this is not required. In some instances, the more operational components robot <NUM> has, the more degrees of freedom of movement it may have.

As used herein, "end effector" <NUM> may refer to a variety of tools that may be operated by robot <NUM> in order to accomplish various tasks. For example, some robots may be equipped with an end effector <NUM> that takes the form of a claw with two opposing "fingers" or "digits. " Such as claw is one type of "gripper" known as an "impactive" gripper. Other types of grippers may include but are not limited to "ingressive" (e.g., physically penetrating an object using pins, needles, etc.), "astrictive" (e.g., using suction or vacuum to pick up an object), or "contigutive" (e.g., using surface tension, freezing or adhesive to pick up object). More generally, other types of end effectors may include but are not limited to drills, brushes, force-torque sensors, cutting tools, deburring tools, welding torches, sprayers (e.g., for paint, pesticide, cleaning solutions, or other fluids), containers, trays, and so forth. In some implementations, end effector <NUM> may be removable, and various types of modular end effectors may be installed onto robot <NUM>, depending on the circumstances.

Sensors <NUM> may take various forms, including but not limited to three-dimensional ("3D") laser scanners or other 3D vision sensors (e.g., stereographic cameras used to perform stereo visual odometry) configured to provide depth measurements, 2D cameras, light sensors (e.g., passive infrared), force sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors (also referred to as "distance sensors"), depth sensors, torque sensors, barcode readers, radio frequency identification ("RFID") readers, radars, range finders, accelerometers, gyroscopes, compasses, position coordinate sensors (e.g., global positioning system, or "GPS"), speedometers, edge detectors, and so forth. While sensors <NUM><NUM>-m are depicted as being integral with robot <NUM>, this is not meant to be limiting. In some implementations, sensors <NUM> may be located external to, but may be in direct or indirect communication with, robot <NUM>, e.g., as standalone units or as part of control system <NUM>.

Control system <NUM> may include one or more computing systems connected by one or more networks (not depicted) that control operation of robot <NUM> to various degrees. An example of such a computing system is depicted schematically in <FIG>. In some implementations, control system <NUM> may be operated by a user (not depicted) to exert a relatively high level of control over robot <NUM>, e.g., in real time in response to signals received by a user interface engine <NUM> and/or one or more readings from one or more sensors <NUM>. In other implementations, control system <NUM> exerts less direct control over robot <NUM>. For example, control system <NUM> may provide robot <NUM> with a high level task such as "go to kitchen, clear table, and load dishwasher. " Logic <NUM> on robot <NUM> may convert such high level tasks into robot action, e.g., by translating one or more high level tasks into a plurality of motion primitives executable by robot <NUM>. In some implementations, control system <NUM> may include a display <NUM> (e.g., CRT, LCD, touchscreen, etc.) on which a graphical user interface <NUM> operable to remotely control robot <NUM> may be rendered.

As noted above, control system <NUM> may be considered "local" to robot <NUM>. For example, if robot <NUM> is deployed in a home, control system <NUM> may be implemented in or near the home on one or more home computers (desktop or laptop), tablet computers, smart phones, smart routers, home servers, smart watches, set top boxes, and so forth. Similarly, if robot <NUM> is deployed in a manufacturing and/or commercial setting, control system <NUM> may be implemented on one or more computing devices having one or more of the aforementioned form factors that is in or near the commercial setting. More generally, to be considered "local" to robot <NUM>, a computing device may be in communication with robot <NUM> through one or more personal area networks ("PANs") and/or local area networks ("LANs"). Put another way, resources are considered "local" to robot <NUM> when they are available when robot <NUM> and/or control system <NUM> are "offline," e.g., not connected to the Internet. Of course, techniques described herein are not limited to control systems that are in communication with robot <NUM> using LANs or PANs. Assuming sufficient bandwidth and acceptable latency, techniques described herein may be implemented using a control system <NUM> that is in communication with robot <NUM> over one or more wide area networks ("WAN") such as the Internet.

Control system <NUM> and robot <NUM> may communicate via one or more communication channels <NUM>. Communication channels <NUM> may utilize various wired and/or wired communication technologies typically utilized over short to medium ranges, e.g., in PANs and/or LANs. For example, in some implementations, communication channel <NUM> may include one or more PANs employing technologies such as Bluetooth, Wireless universal serial bus ("USB"), Z-Wave, Zigbee, Infrared Data Association ("IrDA"), INSTEON, and so forth. Additionally or alternatively, in some implementations, communication channel <NUM> may employ one or more technologies typically associated with LANs, such as Wi-Fi (IEEE <NUM>), Ethernet (IEEE <NUM>), and so forth.

Various modules or engines may be implemented as part of control system <NUM> as software, hardware, or any combination of the two. For example, in <FIG>, control system <NUM> includes an object recognition client 122A, the aforementioned display <NUM> and user interface engine <NUM>, and a vision sensor <NUM>. Vision sensor <NUM> may take various forms, such as a 3D laser scanner or other 3D vision sensor (e.g., stereographic camera used to perform stereo visual odometry) configured to provide depth measurements, a 2D camera, and so forth.

While robot <NUM> and control system <NUM> are depicted separately in <FIG>, this is not meant to be limiting. In various implementations, one or more aspects (e.g., modules, engines, etc.) depicted in <FIG> as implemented on one of robot <NUM> or control system <NUM> may be implemented on the other, may be distributed across both, and/or may be distributed across one or both in combination with other components not depicted in <FIG>. For example, robot <NUM> may operate another instance of object recognition client 122B in memory <NUM>, which may compliment, supplement, or may even replace the first instance of object recognition client 122A that operates on control system <NUM>. In some implementations, control system <NUM> may be implemented entirely or in part using logic <NUM> of robot <NUM>.

Remote object recognition system <NUM> may include one or more computing systems connected by one or more networks (not depicted, sometimes referred to as a "cloud") that facilitate object recognition by one or more robots, including robot <NUM>, and/or by one or more control systems <NUM> that are used to control the robots. An example of such a computing system is depicted schematically in <FIG>. Robot <NUM> and/or control system <NUM> is in communication with remote object recognition system <NUM>, e.g., over connection <NUM> and/or <NUM>. Connections <NUM> and/or <NUM> may be implemented using any wired or wireless technologies typically associated with WAN communication, such as through one or more connected LANs, cellular (e.g., <NUM>, <NUM> and beyond), Tl, Ethernet, DSL, and so forth, though this is not meant to be limiting. In other implementations, connections <NUM> and/or <NUM> may be implemented using LAN or PAN technology. According to the claimed subject matter, the remote object computing system is in wireless communication with the robot.

Various modules or engines may be implemented as part of remote object recognition system <NUM> as software, hardware, or any combination of the two. For example, in <FIG>, remote object recognition system <NUM> includes a root object recognition server <NUM> and a library <NUM> of targeted object recognition modules <NUM><NUM>-J. Library <NUM> may come in the form of one or more databases or other similar data structures suitable for storing targeted object recognition modules <NUM>. Root object recognition server <NUM> may have access to library <NUM> and, as will be discussed in more detail below, may be configured to select one or more targeted object recognition modules <NUM> to process data indicative of an object observed in an environment in which robot <NUM> operates, and to provide (e.g., download to) control system <NUM> and/or robot <NUM> with the selected one or more targeted object recognition modules <NUM>. In some implementations, root object recognition server <NUM> may be operated in whole or in part on robot <NUM> and/or control system <NUM>.

Targeted object recognition modules <NUM><NUM>-J may take various forms. In some implementations, targeted object recognition modules <NUM> may include object models (e.g., computer-aided design, or "CAD", based) that are used and/or triggered to provide inferences about object types/poses, e.g., using vision and/or depth data obtained by one or more vision sensors (e.g., <NUM>, <NUM>). In other implementations, targeted object recognition modules <NUM><NUM>-J may take the form of 2D patterns or profiles of objects that may be matched to portions of 2D image data (e.g., video frames) captured by one or more vision sensors (e.g., <NUM>, <NUM>). In yet other implementations, targeted object recognition modules <NUM><NUM>-J may include routines (e.g., state machines) that may be implemented/triggered by object recognition client <NUM> (122A or 122B) to provide inferences about object type and/or pose.

Object recognition client <NUM> (hereinafter, any operation described as performed by object recognition client <NUM> may be performed by 122A and/or 122B) may be configured to obtain, from one or more vision sensors (e.g., <NUM> or <NUM>), vision data capturing at least a portion of an environment in which robot <NUM> operates. The sensor data may reveal one or more observed objects in the environment with object types and poses that may be unknown.

In various implementations, object recognition client <NUM> is configured to download, from remote object recognition system <NUM>, one or more targeted object recognition modules <NUM> that will be used/applied to make one or more inferences about the one or more observed object. In various implementations, each downloaded targeted object recognition module <NUM> facilitates determination of an object type and/or pose by the object recognition client <NUM>. The one or more downloaded targeted object modules may be selected, e.g., by root object recognition server <NUM> and/or object recognition client <NUM>, from library <NUM> of targeted object recognition modules <NUM> based on various signals.

For example, in some implementations, targeted object recognition modules <NUM> may be selected, e.g., by root object recognition server <NUM> and/or object recognition client <NUM>, for download to object recognition client <NUM> based on data indicative of one or more observed objects. In some implementations, the data indicative of the one or more observed objects may include at least a subset of the sensor data. For example, in some implementations, one or more sensors <NUM> and/or vision sensor <NUM> may provide a so-called "point cloud" that includes, for instance, a color value and depth for each observed point. Object recognition client <NUM> may provide all or a selected subset of the point cloud to root object recognition server <NUM>. The same may be true for 2D sensor data.

In other implementations, the data indicative of the observed one or more objects may include a so-called "soft classifier. " For example, object recognition client <NUM> may include functionality to approximately classify an object's type and/or pose using its limited resources, e.g., with a relatively low level of confidence. In some implementations, object recognition client <NUM> may calculate a soft classifier for an observed object based on data other than observed attributes of the object, such as a location of the object and/or robot <NUM>. For instance, object recognition client <NUM> may be more likely to guess that an observed object is a tool if the object is located in a garage. A robot's location may be determined in various ways, such as using global position system ("GPS") coordinates, inertial measurement units ("IMU"), or various triangulation techniques that leverage one or more wireless connections of robot <NUM>. Based on the soft classifier, root object recognition server <NUM> and/or object recognition client <NUM> may select one or more targeted object recognition modules <NUM> from library <NUM>.

In some implementations, object recognition client <NUM> may provide root object recognition server <NUM> with multiple soft-classifiers for a particular observed object (e.g., multiple object types or poses that the object could potentially match), and root object recognition server <NUM> may select multiple targeted object recognition modules <NUM> that correspond to the multiple soft-classifiers. In some implementations, object recognition client <NUM> may provide root object recognition server <NUM> with other "clues" in addition to or instead of soft-classifiers. For example, object recognition client <NUM> may provide root object recognition server <NUM> with a location of the observed object or robot <NUM> (which may be determined as described above). If root object recognition server <NUM> determines from the received location data that robot <NUM> or the observed object is in a garage, root object recognition server <NUM> may be more likely to select targeted object recognition modules <NUM> that are configured to provide inferences about types and/or poses of tools. If root object recognition server <NUM> determines from the received location data that robot <NUM> or the observed object is in a kitchen, root object recognition server <NUM> may be more likely to select targeted object recognition modules <NUM> that are configured to provide inferences about types and/or poses of items typically found in kitchens, such as cutlery, dishware, etc..

Additionally or alternatively, in various implementations, object recognition client <NUM> may provide information about a task to be performed by robot <NUM> to root object recognition server <NUM>. Based on this task information, root object recognition server <NUM> may select one or more targeted object recognition modules <NUM> for download to object recognition client <NUM>. Suppose a task to be performed by robot <NUM> is "Go to kitchen, clear table, and load dishwasher. " Upon learning of this task, root object recognition server <NUM> may select targeted object recognition modules <NUM> that are geared towards identification of object types and/or poses of objects that are expected to be encountered in a kitchen, such as dishware, cutlery, etc..

Each targeted object recognition module <NUM> that is selected by root object recognition server <NUM> (and/or by object recognition client <NUM>) is used and/or applied to process the data indicative of the observed object(s) in the environment of robot <NUM> in various ways. As was noted above, in some implementations, each downloaded targeted object recognition module <NUM> may be a self-contained state machine that can be triggered, provided with input and then can provide output (e.g., an inference about a pose and/or object type of an observed object).

In some implementations, each targeted object recognition module <NUM> may include an object model associated with a particular object type and/or pose about which the module is configured to provide one or more inferences. Such targeted object recognition modules may be self-contained state machines, or may simply be models that are utilized by other processes (e.g., object recognition client <NUM>) that provide inferences based on the models and data indicative of observed objects provided by object recognition client <NUM>.

However targeted object recognition modules <NUM> are used to process the data indicative of the observed object(s), object recognition client <NUM> determines from output of one or more of the plurality of targeted object recognition modules <NUM>, one or more inferences about an object type or pose of the observed object(s). Based on the one or more inferences, object recognition client <NUM> determines an object type or pose of the observed object.

In some implementations, root object recognition server <NUM> and/or object recognition client <NUM> may learn over time which targeted object recognition modules <NUM> are most likely to be useful to robot <NUM>. For example, the first time robot <NUM> performs a particular task in a particular area, a relatively large number of targeted object recognition modules <NUM> may be selected by root object recognition server <NUM> and downloaded to robot <NUM>, e.g., so that robot <NUM> has a relatively good chance of identifying any objects it encounters. But as robot <NUM> performs the same task (or similar tasks) over time in the same or similar area, robot <NUM> may tend to encounter the same types of objects repeatedly. Accordingly, targeted object recognition modules <NUM> associated with repeatedly-encountered objects may be favored over targeted object recognition modules <NUM> associated with less-often-encountered objects. For example, object recognition client <NUM> may track how many times a particular targeted object recognition module <NUM> is used to determine an observed object's type or pose. This tracking information may be made available to root object recognition server <NUM>, so that in the future, root object recognition server <NUM> may more intelligently select targeted object recognition modules <NUM> from library <NUM>. Additionally or alternatively, object recognition client <NUM> may use this tracking information to request, e.g., from root object recognition server <NUM>, specific targeted object recognition modules <NUM>.

<FIG> depicts an example scenario in which an environment in which a robot (not depicted) operates includes a table <NUM> with three objects <NUM>A-C on top. The environment appears to be a kitchen or dining area based on the fact that the three objects <NUM>A-C take the form of a plate, a cup, and a bowl. In this example, a user (not depicted) controls the robot using a control system <NUM> in the form of a tablet computer. Control system <NUM> includes a display <NUM> in the form of a touchscreen, and at least a front-facing camera (not visible) that is currently capturing in its field of view table <NUM> and objects <NUM>A-C. A graphical user interface <NUM> is rendered on display <NUM> that depicts the field of view of the camera, and hence depicts table <NUM> and objects <NUM>A-C.

Using techniques described herein, control system <NUM>, e.g., by way of an object recognition client (not depicted in <FIG>, see 122A in <FIG>) executing thereon, has downloaded and obtained inferences from a plurality of targeted object recognition modules (<NUM><NUM>-J in <FIG>) about object types of objects <NUM>A-C. These inferences are displayed as part of graphical user interface <NUM>. For example, first object <NUM>A has been inferred to be either a plate or a bowl. These inferences are accompanied by confidence measures that indicate how confident the respective targeted object recognition module <NUM> is about its inference. For example, one targeted object recognition module <NUM> configured to identify plates has identified first object <NUM>A as a plate with a confidence measure of <NUM> (out of <NUM>). Another targeted object recognition module <NUM> configured to identify bowls has identified first object <NUM>A as a bowl with a much lower confidence measure of <NUM>. Based on these confidence measures, object recognition client 122A (or in some scenarios, root object recognition server <NUM>) may determine that first object <NUM>A is most likely a plate.

While confidence measures depicted in <FIG> are within the range of <NUM>-<NUM>, this is not meant to be limiting. Confidence measures may fall within various types of ranges, such as <NUM>-<NUM>, or any other range. And while the inferences and respective confidence measures are visibly rendered on display <NUM>, this is not required. In many cases, these inferences and confidence measures may be used "under the hood" by object recognition client <NUM> (122A on control system <NUM>/<NUM> or 122B on robot <NUM>) and/or root object recognition server <NUM> to determine object types/poses, without displaying anything to a user. Graphical user interface <NUM> of <FIG> may be used, for instance, to debug robot operation and/or to demonstrate to a user how objects are being classified, and is being used here to demonstrate disclosed techniques.

Second object <NUM>B in <FIG> has been alternately inferred to be a cup (with a confidence measure of <NUM>) and a glass (with a confidence measure of <NUM>). In some implementations, because the confidence value associated with the inference of cup is slightly greater than the confidence value associated with the inference of glass, object recognition client <NUM> may simply determine that second object <NUM>B is a cup. However, because the confidence measures are so close, in some implementations, object recognition client <NUM> may take additional actions to attempt to disambiguate between the two conflicting inferences.

For example, in some implementations, additional information may be obtained from one or more sensors <NUM> to attempt to obtain a more accurate inference about second object <NUM>B. In other implementations, other signals or "clues" may be used to disambiguate between conflicting inferences. For example, object recognition client <NUM> may take into account the time of day (e.g., morning) to determine that second object <NUM>B is more likely a paper coffee cup than a glass. In other implementations, object recognition client <NUM> may consult with an object inventory associated with an operator of a robot to determine that the operator does not own a glass matching the "glass" inference, and therefore the "cup" inference must be correct.

In yet other implementations, canonical models associated with each inference, such as computer aided designs ("CAD") associated with cups and glasses, may be used to render one or more canonical cups and glasses. In some instances these canonical objects may be rendered in poses inferred by targeted object recognition modules <NUM> (e.g., in addition to inferred object types). Sensor data depicting second object 252B (e.g., 2D camera data) may then be compared with each rendered canonical model to detect a closest match. The closest match may be used to resolve the conflict between the cup and glass inferences. In some implementations, shapes of the canonical models may be used to disambiguate between multiple conflicting inferences. In some implementations, other characteristics of the canonical models, such as colors, opaqueness, transparency, reflectiveness, etc., may be used to disambiguate between multiple conflicting inferences. For example, a canonical model of a paper coffee cup may indicate that the cup is opaque and/or has a flat (i.e. "matte"), nonreflective surface. By contrast, a canonical model of a glass may indicate that the glass is transparent and/or has a reflective surface.

Third object <NUM>C in <FIG> has been alternately inferred to be a bowl (with a confidence measure of <NUM>) and a hat (with a confidence measure of <NUM>). While not depicted in <FIG>, the "hat" inference may be coupled with an inference about its pose, such as that the hat is upside down. In some implementations, because the confidence value associated with the inference of "bowl" is much greater than the confidence value associated with the inference of "hat," object recognition client <NUM> may simply determine that third object <NUM>C is a bowl.

<FIG> demonstrates example scenarios in which root object recognition server <NUM> (or object recognition client <NUM>) may select one or more targeted object recognition modules <NUM> from library <NUM> based on various signals. A robot <NUM> is depicted in the form of a mobile robot arm with a "claw" style end effector <NUM> and an onboard vision sensor <NUM>. Suppose a user operates control system <NUM> (e.g., her smartphone or tablet computer, not depicted in <FIG>) to instruct robot <NUM> to go into the kitchen <NUM> and perform some kitchen-related task (e.g., empty dishwasher, clean floor, etc.). In some implementations, data indicative of this task may be provided to root object recognition server <NUM>. Root object recognition server <NUM> may in turn select one or more targeted object recognition modules <NUM> that facilitate inference of object types or poses of objects likely to be encountered in kitchen <NUM>, and may download the selected targeted object recognition modules <NUM> to robot <NUM> and/or to the undepicted control system (<NUM> in <FIG>). Object recognition client <NUM> may then utilize these downloaded targeted object recognition modules <NUM> to determine object types and/or poses associated with one or more observed objects in kitchen <NUM>.

A wireless access point <NUM> is depicted in kitchen <NUM>. Wireless access point <NUM> may provide a strong, reliable signal within kitchen <NUM>. However, wireless access point <NUM> may not provide as strong or reliable a signal in other areas, such as on an outdoor porch <NUM>. Suppose the user instructs robot <NUM> to go to porch <NUM> and perform some porch-related task (e.g., water flowers, waterproof porch, etc.). According to the claimed embodiment, object recognition client <NUM> operates on robot <NUM> (e.g., 122B in <FIG>). Object recognition client 122B may determine, e.g., based on past instances in which robot <NUM> was operated on porch <NUM>, that the wireless signal on porch <NUM> is weak and/or unreliable. Accordingly, prior to robot <NUM> travelling to porch <NUM> to perform its task (e.g., while robot <NUM> travels through kitchen <NUM> towards porch <NUM>), object recognition client 122B may notify root object recognition server <NUM> of the task and/or the location of the task. As discussed above, root object recognition server <NUM> may select and download to object recognition client 122B one or more targeted object recognition modules <NUM>. Object recognition client 122B then uses these downloaded modules to determine object types and/or poses of objects encountered by robot <NUM> on porch <NUM>.

In some instances, the targeted object recognition modules <NUM> will be downloaded to robot <NUM> while robot <NUM> travels through kitchen <NUM> to porch <NUM>, e.g., while robot <NUM> is in range of wireless access point <NUM>. However, robot <NUM> may reach the door between kitchen <NUM> and porch <NUM> (and hence, become potentially out-of-range of wireless access point <NUM>) before the selected targeted object recognition modules <NUM> are downloaded to robot <NUM>. According to the claimed subject matter, robot <NUM> is operated to enter a second area from a first area after determining that the downloading of one or more targeted object recognition modules is complete, where a wireless signal provided by a wireless access point (<NUM>) in the second area is less strong and/or less reliable than in the first area. For instance, robot <NUM> may pause while still in range of wireless access point <NUM> until robot <NUM> is able to fully download targeted objects recognition modules <NUM> it will need on porch <NUM>.

Referring now to <FIG>, an example method <NUM> of downloading targeted object recognition modules that are selected from a library of candidate targeted object recognition modules based on various signals is described. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems. For instance, some operations may be performed at robot <NUM>, while other operations may be performed by one or more components of control system <NUM> and/or remote object recognition system <NUM>. Moreover, while operations of method <NUM> are shown in a particular order, this is not meant to be limiting.

At block <NUM>, the system operates an object recognition client on robot <NUM> (e.g., 122B), to perform and/or otherwise facilitate object recognition for one or more robots. At block <NUM>, the system (e.g., root object recognition server <NUM> or object recognition client <NUM>) selects one or more targeted object recognition modules (<NUM>) from a library (e.g., <NUM>) of object recognition modules based on various signals. As described above, these signals include a task assigned to the robot. For example, if the robot is instructed to clean a child's room, then targeted object recognition modules associated with objects likely to be found in a child's room, such as toys, may be selected. If the robot is instructed to play a board game, then targeted object recognition modules associated with game pieces used in that board game may be selected.

Other signals may be used as well. For example, one or more available resources of the robot may be considered. In some implementations, object recognition client <NUM> may notify root object recognition server <NUM> of its limited resources. For example, object recognition client <NUM> may explicitly state in its request for targeted object recognition modules that robot <NUM> has limited resources. Additionally or alternatively, object recognition client <NUM> may implicitly indicate limited resources by, for instance, requesting that root object recognition server <NUM> limit the number of targeted object recognition modules it selects and downloads to object recognition client <NUM>.

One resource available to a robot is a wireless signal available to the robot, e.g., such as Wi-Fi signal, a cellular signal, a Bluetooth signal, and so forth. Suppose a robot is operating in an area with a weak or unreliable signal, or a signal with limited bandwidth. Before such a robot performs a task, it may notify root object recognition server <NUM> of the suboptimal wireless signal. Root object recognition server <NUM> may take this into account when selecting targeted object recognition modules <NUM> to download to the robot. In some implementations, root object recognition server <NUM> may be more particular when selecting targeted object recognition modules to download to a robot with a low-bandwidth connection. For example, under normal circumstances (e.g., a strong wireless signal with ample bandwidth), root object recognition server <NUM> may select and download to the robot a relatively large number of targeted object recognition modules associated with various types objects, some associated with objects of which there may only be a slight chance the robot will encounter while performing a task. However, when the robot has a low-bandwidth connection, root object recognition server <NUM> may only select a smaller set of targeted object recognition modules to download to the robot, such as the n modules associated with objects that the robot is most likely to encounter. In various implementations, the number of selected modules n may be set by a user and/or determined dynamically, e.g., based on bandwidth available to the robot.

Other available resources of the robot may be considered as well when selecting targeted object recognition modules, in addition to or instead of attributes of a wireless signal available to the robot. Suppose a robot has limited processing power. Providing an object recognition client 122B operating on the robot with too many targeted object recognition modules <NUM> may overburden the robot's processing capabilities, causing the robot to stall or behave sluggishly. Accordingly, in some implementations, root object recognition server <NUM> may be more discerning when selecting targeted object recognition modules to download to a robot with limited processing power. For example, and similar to above, root object recognition server <NUM> may only select a relatively small set of targeted object recognition modules to download to the robot, such as the n modules associated with objects that the robot is most likely to encounter. Root object recognition server <NUM> may be similarly discerning when selected targeted object recognition modules to download to a robot having limited battery power.

Other signals or "clues" may be considered as well when selecting targeted object recognition modules to download to object recognition client <NUM> at block <NUM>. For example, and as was mentioned above, data indicative of one or more unclassified observed objects in the environment may be provided to root object recognition server <NUM>, which may in turn use this data to select targeted object recognition modules. As was noted above, this data indicative of the one or more unclassified observed objects may include, for instance, at least a subset of vision data obtained by one or more sensors (in which case operations of block <NUM> described below may be performed prior to or concurrently with operations of block <NUM>), one or more soft classifiers, one or more other signals, or "clues," about objects likely to be encountered by the robot when performing its task, and so forth.

Referring back to <FIG>, at block <NUM>, the system downloads to the object recognition client <NUM> (which operates on the robot or, in an example that is useful for understanding the claimed subject matter, on a local control system used to control the robot) the targeted object recognition modules selected at block <NUM>. In some implementations, the downloaded targeted object recognition modules may take the form of low level machine instructions compiled to execute on a specific computing architecture of the robot or control system. In some implementations, the downloaded targeted object recognition modules may take the form of higher-level instructions that can be, for instance, interpreted by object recognition client <NUM> to determine object types or poses. For example, in some implementations, the downloaded targeted object recognition modules may be constructed using markup languages such as the extensible markup language ("XML") or the Unified Modeling Language ("UML") to convey a state machine, and object recognition client <NUM> may implement these state machines.

In some implementations, the downloaded targeted object recognition modules may take the form of a CAD model, 2D model, and/or associated annotations, which may be compared by object recognition client <NUM> to sensor data indicative of observed objects in the environment. In some implementations, the downloaded targeted object recognition modules may take the form of machine learning classifiers that are trained to label object types and/or poses of particular objects.

At block <NUM>, the system provides, e.g., to object recognition client <NUM>, vision data capturing the robot environment. For example, a 3D vision sensor mounted on the robot or nearby may obtain a point cloud of data. This point cloud (or a subset thereof) may be provided to object recognition client <NUM>. As another example, one or more 2D images may be captured of the environment and provided to object recognition client <NUM>.

Based on the vision data, at block <NUM>, the system utilizes the one or more targeted object recognition modules downloaded at block <NUM> to determine information about one or more observed objects in the environment, such as an object type and/or pose. For example, a targeted object recognition module configured to identify cups may receive, as input, point cloud data obtained at block <NUM>, and provide, as output, an inference that an observed object is a cup and/or a current pose of the cup (e.g., orientation, right-side up, upside down, on its side, full of fluid, cap attached, etc.). In some instances, the targeted object recognition module may also output a measure of confidence associated with its inference, as was discussed previously with regard to <FIG>. Based on one or more object type/pose inferences obtained using downloaded targeted object recognition modules, the system determines information about one or more observed objects in the environment, such as an object type and/or pose.

Storage subsystem <NUM> stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem <NUM> may include the logic to perform selected aspects of method <NUM>, and/or to implement one or more aspects of robot <NUM>, control system <NUM>, or remote object recognition system <NUM>. A file storage subsystem <NUM> can provide persistent storage for program and data files, and may include a hard disk drive, a CD-ROM drive, an optical drive, or removable media cartridges. Modules implementing the functionality of certain implementations may be stored by file storage subsystem <NUM> in the storage subsystem <NUM>, or in other machines accessible by the processor(s) <NUM>.

Claim 1:
A computer-implemented method comprising:
operating, by one or more processors of a robot (<NUM>), an object recognition client (<NUM>) operating on the robot to facilitate object recognition for the robot (<NUM>);
determining, by the object recognition client, that a wireless signal provided by a wireless access point (<NUM>) in a second area where the robot (<NUM>) is required to travel, from a first area, to perform a task is less strong and/or less reliable than in the first area;
before operating the robot to enter the second area, downloading, by the object recognition client, from a remote computing system (<NUM>) in wireless communication with the robot, one or more targeted object recognition modules (<NUM><NUM> to <NUM>i), wherein each targeted object recognition module is usable to provide inference of an object type or pose of an observed object, and wherein the one or more targeted object recognition modules are selected from a library of targeted object recognition modules (<NUM>) based on the task to be performed by the robot in the second area;
determining that the downloading is complete;
after determining that the downloading is complete, operating the robot to enter the second area;
obtaining, by the object recognition client, from one or more vision sensors (<NUM>), vision data capturing at least a portion of the second area; and
determining, by the object recognition client, based on the vision data and the one or more downloaded targeted object recognition modules, information about an object observed in the second area.