Patent Publication Number: US-2023133480-A1

Title: Thin object detection and avoidance in aerial robots

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     The present application claims the benefit of U.S. Provisional Patent Application 63/274,450, filed on Nov. 1, 2021, which is hereby incorporated by reference in its entirety. 
    
    
     TECHNICAL FIELD 
     The disclosure generally relates to detecting objects by robots and, more specifically, to robots that use neural networks to detect and avoid thin objects. 
     BACKGROUND 
     For aerial robots such as drones to be autonomous, aerial robots need to navigate through the environment without colliding with objects. Certain objects are more difficult to detect by the sensors of the robot due to the objects&#39; sizes and shapes. For example, even the state-of-the-art robots are unable to detect any electrical wires or other cables because those wires are often too thin for the robots to generate point cloud data with depth measurements of the wires. Without manual control, aerial robots often collide with those wires, causing damages to property and creating potentially dangerous situations. 
     SUMMARY 
     Embodiments relate to aerial robots that include image sensors for capturing images of environments. The aerial robot receives a first image of an environment captured at a first location. The aerial robot identifies an object in the first image. The object may be a thin object. The aerial robot identifies one or more first pixels in the first image that correspond to one or more targeted features of the identified object. The aerial robot receives a second image of the environment captured at the second location. The aerial robot receives its distance data that estimates the movement of the aerial robot from the first location to the second location. The aerial robot identifies one or more second pixels in the second image that correspond to the targeted features of the object as appeared in the second image. The aerial robot determines an estimated distance between the aerial robot and the object based on the changes of locations of the second pixels from the first pixels relative to the movement of the aerial robot provided by the distance data. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a block diagram that illustrates a system environment of an example storage site, in accordance with some embodiments. 
         FIG.  2    is a block diagram that illustrates components of an example robot and an example base station, in accordance with some embodiments. 
         FIG.  3    is a flowchart that depicts an example process for managing the inventory of a storage site, in accordance with some embodiments. 
         FIG.  4    is a conceptual diagram of an example layout of a storage site that is equipped with a robot, in accordance with some embodiments. 
         FIG.  5    is a flowchart depicting an example navigation process of a robot, in accordance with some embodiments. 
         FIG.  6    is a conceptual diagram illustrating a robot detecting a thin object, in accordance with some embodiments. 
         FIG.  7    is a flowchart depicting an example process of a robot for detecting thin objects in the environment, in accordance with some embodiments. 
         FIG.  8 A  is a block diagram illustrating an example machine learning model, in accordance with some embodiments. 
         FIG.  8 B  is a block diagram illustrating another example machine learning model, in accordance with some embodiments. 
         FIG.  9    is a block diagram illustrating components of an example computing machine, in accordance with some embodiments. 
     
    
    
     The figures depict, and the detailed description describes, various non-limiting embodiments for purposes of illustration only. 
     DETAILED DESCRIPTION 
     The figures (FIGS.) and the following description relate to preferred embodiments by way of illustration only. One of skill in the art may recognize alternative embodiments of the structures and methods disclosed herein as viable alternatives that may be employed without departing from the principles of what is disclosed. 
     Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying figures. It is noted that wherever practicable similar or like reference numbers may be used in the figures and may indicate similar or like functionality. The figures depict embodiments of the disclosed system (or method) for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein. 
     Embodiments relate to aerial robots that navigate through environments by using machine learning models to identify thin objects in the environments and estimate the distances between the robots and the identified thin objects. A robot may include a thin object detector that uses a convolutional neural network that distinguishes pixels corresponding to the thin object from the rest of the pixels. The robot may detect identifiable features of the thin object and tracks the movement of the identified features over different image frames captured by the image sensor of the robot. Based on the movement of the robot and pose evaluation that may be generated by a state estimator and an inertial measurement unit, the robot analyzes the movement of the identified features appeared in the images and determines the estimated distance between the robot and the thin object. 
     System Overview 
     FIG. ( FIG.  1    is a block diagram that illustrates a system environment  100  of an example robotically-assisted or fully autonomous storage site, in accordance with some embodiments. By way of example, the system environment  100  includes a storage site  110 , a robot  120 , a base station  130 , an inventory management system  140 , a computing server  150 , a data store  160 , and a user device  170 . The entities and components in the system environment  100  communicate with each other through the network  180 . In various embodiments, the system environment  100  may include different, fewer, or additional components. Also, while each of the components in the system environment  100  is described in a singular form, the system environment  100  may include one or more of each of the components. For example, the storage site  110  may include one or more robots  120  and one or more base stations  130 . Each robot  120  may have a corresponding base station  130  or multiple robots  120  may share a base station  130 . 
     A storage site  110  may be any suitable facility that stores, sells, or displays inventories such as goods, merchandise, groceries, articles and collections. Example storage sites  110  may include warehouses, inventory sites, bookstores, shoe stores, outlets, other retail stores, libraries, museums, etc. A storage site  110  may include a number of regularly shaped structures. Regularly shaped structures may be structures, fixtures, equipment, furniture, frames, shells, racks, or other suitable things in the storage site  110  that have a regular shape or outline that can be readily identifiable, whether the things are permanent or temporary, fixed or movable, weight-bearing or not. The regularly shaped structures are often used in a storage site  110  for storage of inventory. For example, racks (including metallic racks, shells, frames, or other similar structures) are often used in a warehouse for the storage of goods and merchandise. However, not all regularly shaped structures may need to be used for inventory storage. A storage site  110  may include a certain layout that allows various items to be placed and stored systematically. For example, in a warehouse, the racks may be grouped by sections and separated by aisles. Each rack may include multiple pallet locations that can be identified using a row number and a column number. A storage site may include high racks and low racks, which may, in some case, largely carry most of the inventory items near the ground level. 
     A storage site  110  may include one or more robots  120  that are used to keep track of the inventory and to manage the inventory in the storage site  110 . For the ease of reference, the robot  120  may be referred to in a singular form, even though more than one robot  120  may be used. Also, in some embodiments, there can be more than one type of robot  120  in a storage site  110 . For example, some robots  120  may specialize in scanning inventory in the storage site  110 , while other robots  120  may specialize in moving items. A robot  120  may also be referred to as an autonomous robot, an inventory cycle-counting robot, an inventory survey robot, an inventory detection robot, or an inventory management robot. An inventory robot may be used to track inventory items, move inventory items, and carry out other inventory management tasks. The degree of autonomy may vary from embodiments to embodiments. For example, in some embodiments, the robot  120  may be fully autonomous so that the robot  120  automatically performs assigned tasks. In another embodiment, the robot  120  may be semi-autonomous such that it can navigate through the storage site  110  with minimal human commands or controls. In some embodiments, no matter what the degree of autonomy it has, a robot  120  may also be controlled remotely and may be switched to a manual mode. The robot  120  may take various forms such as an aerial drone, a ground robot, a vehicle, a forklift, and a mobile picking robot. 
     A base station  130  may be a device for the robot  120  to return and, for an aerial robot, to land. The base station  130  may include more than one return site. The base station  130  may be used to repower the robot  120 . Various ways to repower the robot  120  may be used in different embodiments. For example, in some embodiments, the base station  130  serves as a battery-swapping station that exchanges batteries on a robot  120  as the robot arrives at the base station to allow the robot  120  to quickly resume duty. The replaced batteries may be charged at the base station  130 , wired or wirelessly. In another embodiment, the base station  130  serves as a charging station that has one or more charging terminals to be coupled to the charging terminal of the robot  120  to recharge the batteries of the robot  120 . In yet another embodiment, the robot  120  may use fuel for power and the base station  130  may repower the robot  120  by filling its fuel tank. 
     The base station  130  may also serve as a communication station for the robot  120 . For example, for certain types of storage sites  110  such as warehouses, network coverage may not be present or may only be present at certain locations. The base station  130  may communicate with other components in the system environment  100  using wireless or wired communication channels such as Wi-Fi or an Ethernet cable. The robot  120  may communicate with the base station  130  when the robot  120  returns to the base station  130 . The base station  130  may send inputs such as commands to the robot  120  and download data captured by the robot  120 . In embodiments where multiple robots  120  are used, the base station  130  may be equipped with a swarm control unit or algorithm to coordinate the movements among the robots. The base station  130  and the robot  120  may communicate in any suitable ways such as radio frequency, Bluetooth, near-field communication (NFC), or wired communication. While, in some embodiments, the robot  120  mainly communicates to the base station, in other embodiments the robot  120  may also have the capability to directly communicate with other components in the system environment  100 . In some embodiments, the base station  130  may serve as a wireless signal amplifier for the robot  120  to directly communicate with the network  180 . 
     The inventory management system  140  may be a computing system that is operated by the administrator (e.g., a company that owns the inventory, a warehouse management administrator, a retailer selling the inventory) using the storage site  110 . The inventory management system  140  may be a system used to manage the inventory items. The inventory management system  140  may include a database that stores data regarding inventory items and the items&#39; associated information, such as quantities in the storage site  110 , metadata tags, asset type tags, barcode labels and location coordinates of the items. The inventory management system  140  may provide both front-end and back-end software for the administrator to access a central database and point of reference for the inventory and to analyze data, generate reports, forecast future demands, and manage the locations of the inventory items to ensure items are correctly placed. An administrator may rely on the item coordinate data in the inventory management system  140  to ensure that items are correctly placed in the storage site  110  so that the items can be readily retrieved from a storage location. This prevents an incorrectly placed item from occupying a space that is reserved for an incoming item and also reduces time to locate a missing item at an outbound process. 
     The computing server  150  may be a server that is tasked with analyzing data provided by the robot  120  and provide commands for the robot  120  to perform various inventory recognition and management tasks. The robot  120  may be controlled by the computing server  150 , the user device  170 , or the inventory management system  140 . For example, the computing server  150  may direct the robot  120  to scan and capture pictures of inventory stored at various locations at the storage site  110 . Based on the data provided by the inventory management system  140  and the ground truth data captured by the robot  120 , the computing server  150  may identify discrepancies in two sets of data and determine whether any items may be misplaced, lost, damaged, or otherwise should be flagged for various reasons. In turn, the computing server  150  may direct a robot  120  to remedy any potential issues such as moving a misplaced item to the correct position. In some embodiments, the computing server  150  may also generate a report of flagged items to allow site personnel to manually correct the issues. 
     The computing server  150  may include one or more computing devices that operate at different locations. For example, a part of the computing server  150  may be a local server that is located at the storage site  110 . The computing hardware such as the processor may be associated with a computer on site or may be included in the base station  130 . Another part of the computing server  150  may be a cloud server that is geographically distributed. The computing server  150  may serve as a ground control station (GCS), provide data processing, and maintain end-user software that may be used in a user device  170 . A GCS may be responsible for the control, monitor and maintenance of the robot  120 . In some embodiments, GCS is located on-site as part of the base station  130 . The data processing pipeline and end-user software server may be located remotely or on-site. 
     The computing server  150  may maintain software applications for users to manage the inventory, the base station  130 , and the robot  120 . The computing server  150  and the inventory management system  140  may or may not be operated by the same entity. In some embodiments, the computing server  150  may be operated by an entity separated from the administrator of the storage site. For example, the computing server  150  may be operated by a robotic service provider that supplies the robot  120  and related systems to modernize and automate a storage site  110 . The software application provided by the computing server  150  may take several forms. In some embodiments, the software application may be integrated with or as an add-on to the inventory management system  140 . In another embodiment, the software application may be a separate application that supplements or replaces the inventory management system  140 . In some embodiments, the software application may be provided as software as a service (SaaS) to the administrator of the storage site  110  by the robotic service provider that supplies the robot  120 . 
     The data store  160  includes one or more storage units such as memory that takes the form of non-transitory and non-volatile computer storage medium to store various data that may be uploaded by the robot  120  and inventory management system  140 . For example, the data stored in data store  160  may include pictures, sensor data, and other data captured by the robot  120 . The data may also include inventory data that is maintained by the inventory management system  140 . The computer-readable storage medium is a medium that does not include a transitory medium such as a propagating signal or a carrier wave. The data store  160  may take various forms. In some embodiments, the data store  160  communicates with other components by the network  180 . This type of data store  160  may be referred to as a cloud storage server. Example cloud storage service providers may include AWS, AZURE STORAGE, GOOGLE CLOUD STORAGE, etc. In another embodiment, instead of a cloud storage server, the data store  160  is a storage device that is controlled and connected to the computing server  150 . For example, the data store  160  may take the form of memory (e.g., hard drives, flash memories, discs, ROMs, etc.) used by the computing server  150  such as storage devices in a storage server room that is operated by the computing server  150 . 
     The user device  170  may be used by an administrator of the storage site  110  to provide commands to the robot  120  and to manage the inventory in the storage site  110 . For example, using the user device  170 , the administrator can provide task commands to the robot  120  for the robot to automatically complete the tasks. In one case, the administrator can specify a specific target location or a range of storage locations for the robot  120  to scan. The administrator may also specify a specific item for the robot  120  to locate or to confirm placement. Examples of user devices  170  include personal computers (PCs), desktop computers, laptop computers, tablet computers, smartphones, wearable electronic devices such as smartwatches, or any other suitable electronic devices. 
     The user device  170  may include a user interface  175 , which may take the form of a graphical user interface (GUI). Software application provided by the computing server  150  or the inventory management system  140  may be displayed as the user interface  175 . The user interface  175  may take different forms. In some embodiments, the user interface  175  is part of a front-end software application that includes a GUI displayed at the user device  170 . In one case, the front-end software application is a software application that can be downloaded and installed at user devices  170  via, for example, an application store (e.g., App Store) of the user device  170 . In another case, the user interface  175  takes the form of a Web interface of the computing server  150  or the inventory management system  140  that allows clients to perform actions through web browsers. In another embodiment, user interface  175  does not include graphical elements but communicates with the computing server  150  or the inventory management system  140  via other suitable ways such as command windows or application program interfaces (APIs). 
     The communications among the robot  120 , the base station  130 , the inventory management system  140 , the computing server  150 , the data store  160 , and the user device  170  may be transmitted via a network  180 , for example, via the Internet. In some embodiments, the network  180  uses standard communication technologies and/or protocols. Thus, the network  180  can include links using technologies such as Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 3G, 4G, LTE, 5G, digital subscriber line (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express, etc. Similarly, the networking protocols used on the network  180  can include multiprotocol label switching (MPLS), the transmission control protocol/Internet protocol (TCP/IP), the user datagram protocol (UDP), the hypertext transport protocol (HTTP), the simple mail transfer protocol (SMTP), the file transfer protocol (FTP), etc. The data exchanged over the network  180  can be represented using technologies and/or formats including the hypertext markup language (HTML), the extensible markup language (XML), etc. In addition, all or some of the links can be encrypted using conventional encryption technologies such as secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), Internet protocol security (IPsec), etc. The network  180  also includes links and packet switching networks such as the Internet. In some embodiments, two computing servers, such as computing server  150  and inventory management system  140 , may communicate through APIs. For example, the computing server  150  may retrieve inventory data from the inventory management system  140  via an API. 
     Example Robot and Base Station 
       FIG.  2    is a block diagram illustrating components of an example robot  120  and an example base station  130 , in accordance with some embodiments. The robot  120  may include an image sensor  210 , a processor  215 , memory  220 , a flight control unit (FCU)  225  that includes an inertial measurement unit (IMU)  230 , a state estimator  235 , a visual reference engine  240 , a planner  250 , a communication engine  255 , an I/O interface  260 , and a power source  265 . The functions of the robot  120  may be distributed among various components in a different manner than described below. In various embodiments, the robot  120  may include different, fewer, and/or additional components. Also, while each of the components in  FIG.  2    is described in a singular form, the components may present in plurality. For example, a robot  120  may include more than one image sensor  210  and more than one processor  215 . 
     The image sensor  210  captures images of an environment of a storage site for navigation, localization, collision avoidance, object recognition and identification, and inventory recognition purposes. A robot  120  may include more than one image sensor  210  and more than one type of such image sensor  210 . For example, the robot  120  may include a digital camera that captures optical images of the environment for the state estimator  235 . For example, data captured by the image sensor  210  may also be provided to the VIO unit  236  that may be included in the state estimator  235  for localization purposes such as to determine the position and orientation of the robot  120  with respect to an inertial frame, such as a global frame whose location is known and fixed. The robot  120  may also include a stereo camera that includes two or more lenses to allow the image sensor  210  to capture three-dimensional images through stereoscopic photography. For each image frame, the stereo camera may generate pixel values such as in red, green, and blue (RGB) and point cloud data that includes depth information. The images captured by the stereo camera may be provided to visual reference engine  240  for object recognition purposes. The image sensor  210  may also be another type of image sensor such as a light detection and ranging (LIDAR) sensor, an infrared camera, and 360-degree depth cameras. The image sensor  210  may also capture pictures of labels (e.g., barcodes) on items for inventory cycle-counting purposes. In some embodiments, a single stereo camera may be used for various purposes. For example, the stereo camera may provide image data to the visual reference engine  240  for object recognition. The stereo camera may also be used to capture pictures of labels (e.g., barcodes). In some embodiments, the robot  120  includes a rotational mount such as a gimbal that allows the image sensor  210  to rotate at different angles and to stabilize images captured by the image sensor  210 . In some embodiments, the image sensor  210  may also capture data along the path for the purpose of mapping the storage site. 
     The robot  120  includes one or more processors  215  and one or more memories  220  that store one or more sets of instructions. The one or more sets of instructions, when executed by one or more processors, cause the one or more processors to carry out processes that are implemented as one or more software engines. Various components, such as FCU  225  and state estimator  235 , of the robot  120  may be implemented as a combination of software and hardware (e.g., sensors). The robot  120  may use a single general processor to execute various software engines or may use separate more specialized processors for different functionalities. In some embodiments, the robot  120  may use a general-purpose computer (e.g., a CPU) that can execute various instruction sets for various components (e.g., FCU  225 , visual reference engine  240 , state estimator  235 , planner  250 ). The general-purpose computer may run on a suitable operating system such as LINUX, ANDROID, etc. For example, in some embodiments, the robot  120  may carry a smartphone that includes an application used to control the robot. In another embodiment, the robot  120  includes multiple processors that are specialized in different functionalities. For example, some of the functional components such as FCU  225 , visual reference engine  240 , state estimator  235 , and planner  250  may be modularized and each includes its own processor, memory, and a set of instructions. The robot  120  may include a central processor unit (CPU) to coordinate and communicate with each modularized component. Hence, depending on embodiments, a robot  120  may include a single processor or multiple processors  215  to carry out various operations. The memory  220  may also store images and videos captured by the image sensor  210 . The images may include images that capture the surrounding environment and images of the inventory such as barcodes and labels. 
     The flight control unit (FCU)  225  may be a combination of software and hardware, such as the inertial measurement unit (IMU)  230  and other sensors, to control the movement of the robot  120 . For ground robot  120 , the flight control unit  225  may also be referred to as a microcontroller unit (MCU). The FCU  225  relies on information provided by other components to control the movement of the robot  120 . For example, the planner  250  determines the path of the robot  120  from a starting point to a destination and provides commands to the FCU  225 . Based on the commands, the FCU  225  generates electrical signals to various mechanical parts (e.g., actuators, motors, engines, wheels) of the robot  120  to adjust the movement of the robot  120 . The precise mechanical parts of the robots  120  may depend on the embodiments and the types of robots  120 . 
     The IMU  230  may be part of the FCU  225  or may be an independent component. The IMU  230  may include one or more accelerometers, gyroscopes, and other suitable sensors to generate measurements of forces, linear accelerations, and rotations of the robot  120 . For example, the accelerometers measure the force exerted on the robot  120  and detect the linear acceleration. Multiple accelerometers cooperate to detect the acceleration of the robot  120  in the three-dimensional space. For instance, a first accelerometer detects the acceleration in the x-direction, a second accelerometer detects the acceleration in the y-direction, and a third accelerometer detects the acceleration in the z-direction. The gyroscopes detect the rotations and angular acceleration of the robot  120 . Based on the measurements, a processor  215  may obtain the estimated localization of the robot  120  by integrating the translation and rotation data of the IMU  230  with respect to time. The IMU  230  may also measure the orientation of the robot  120 . For example, the gyroscopes in the IMU  230  may provide readings of the pitch angle, the roll angle, and the yaw angle of the robot  120 . 
     The state estimator  235  may correspond to a set of software instructions stored in the memory  220  that can be executed by the processor  215 . The state estimator  235  may be used to generate localization information of the robot  120  and may include various sub-components for estimating the state of the robot  120 . For example, in some embodiments, the state estimator  235  may include a visual-inertial odometry (VIO) unit  236  and an height estimator  238 . In other embodiments, other modules, sensors, and algorithms may also be used in the state estimator  235  to determine the location of the robot  120 . 
     The VIO unit  236  receives image data from the image sensor  210  (e.g., a stereo camera) and measurements from IMU  230  to generate localization information such as the position and orientation of the robot  120 . The localization data obtained from the double integration of the acceleration measurements from the IMU  230  is often prone to drift errors. The VIO unit  236  may extract image feature points and tracks the feature points in the image sequence to generate optical flow vectors that represent the movement of edges, boundaries, surfaces of objects in the environment captured by the image sensor  210 . Various signal processing techniques such as filtering (e.g., Wiener filter, Kalman filter, bandpass filter, particle filter) and optimization, and data/image transformation may be used to reduce various errors in determining localization information. The localization data generated by the VIO unit  236  may include an estimate of the pose of the robot  120 , which may be expressed in terms of the roll angle, the pitch angle, and the yaw angle of the robot  120 . 
     The height estimator  238  may be a combination of software and hardware that are used to determine the absolute height and relative height (e.g., distance from an object that lies on the floor) of the robot  120 . The height estimator  238  may include a downward distance sensor  239  that may measure the height relative to the ground or to an object underneath the robot  120 . The distance sensor  239  may be electromagnetic wave based, laser based, optics based, sonar based, ultrasonic based, or another suitable signal based. For example, the distance sensor  239  may be a laser range finder, a lidar range finder, a sonar range finder, an ultrasonic range finder, or a radar. A range finder may include one or more emitters that emit signals (e.g., infrared, laser, sonar, etc.) and one or more sensors that detect the round trip time of the signal reflected by an object. In some embodiments, the robot  120  may be equipped with a single emitter range finder. The height estimator  238  may also receive data from the VIO unit  236  that may estimate the height of the robot  120 , but usually in a less accurate fashion compared to a distance sensor  239 . The height estimator  238  may include software algorithms to combine data generated by the distance sensor  239  and the data generated by the VIO unit  236  as the robot  120  flies over various objects and inventory that are placed on the floor or other horizontal levels. The data generated by the height estimator  238  may be used for collision avoidance and finding a target location. The height estimator  238  may set a global maximum altitude to prevent the robot  120  from hitting the ceiling. The height estimator  238  also provides information regarding how many rows in the rack are below the robot  120  for the robot  120  to locate a target location. The height data may be used in conjunction with the count of rows that the robot  120  has passed to determine the vertical level of the robot  120 . 
     The visual reference engine  240  may correspond to a set of software instructions stored in the memory  220  that can be executed by the processor  215 . The visual reference engine  240  may include various image processing algorithm and location algorithm to determine the current location of the robot  120 , to identify the objects, edges, and surfaces of the environment near the robot  120 , and to determine an estimated distance and orientation (e.g., yaw) of the robot  120  relative to a nearby surface of an object. The visual reference engine  240  may receive pixel data of a series of images and point cloud data from the image sensor  210 . The location information generated by the visual reference engine  240  may include distance and yaw from an object and center offset from a target point (e.g., a midpoint of a target object). 
     The visual reference engine  240  may include one or more algorithms and machine learning models to create image segmentations from the images captured by the image sensor  210 . The image segmentation may include one or more segments that separate the frames (e.g., vertical or horizontal bars of racks) or outlines of regularly shaped structures appearing in the captured images from other objects and environments. The algorithms used for image segmentation may include a convolutional neural network (CNN). In performing the segmentation, other image segmentation algorithms such as edge detection algorithms (e.g., Canny operator, Laplacian operator, Sobel operator, Prewitt operator), corner detection algorithms, Hough transform, and other suitable feature detection algorithms may also be used. 
     The visual reference engine  240  also performs object recognition (e.g., object detection and further analyses) and keeps track of the relative movements of the objects across a series of images. The visual reference engine  240  may track the number of regularly shaped structures in the storage site  110  that are passed by the robot  120 . For example, the visual reference engine  240  may identify a reference point (e.g., centroid) of a frame of a rack and determine if the reference point passes a certain location of the images across a series of images (e.g., whether the reference point passes the center of the images). If so, the visual reference engine  240  increments the number of regularly shaped structures that have been passed by the robot  120 . 
     The robot  120  may use various components to generate various types of location information (including location information relative to nearby objects and localization information). For example, in some embodiments, the state estimator  235  may process the data from the VIO unit  236  and the height estimator  238  to provide localization information to the planner  250 . The visual reference engine  240  may count the number of regularly shaped structures that the robot  120  has passed to determine a current location. The visual reference engine  240  may generate location information relative to nearby objects. For example, when the robot  120  reaches a target location of a rack, the visual reference engine  240  may use point cloud data to reconstruct a surface of the rack and use the depth data from the point cloud to determine more accurate yaw and distance between the robot  120  and the rack. The visual reference engine  240  may determine a center offset, which may correspond to the distance between the robot  120  and the center of a target location (e.g., the midpoint of a target location of a rack). Using the center offset information, the planner  250  controls the robot  120  to move to the target location and take a picture of the inventory in the target location. When the robot  120  changes direction (e.g., rotations, transitions from horizontal movement to vertical movement, transitions from vertical movement to horizontal movement, etc.), the center offset information may be used to determine the accurate location of the robot  120  relative to an object. 
     The planner  250  may correspond to a set of software instructions stored in the memory  220  that can be executed by the processor  215 . The planner  250  may include various routing algorithms to plan a path of the robot  120  as the robot travels from a first location (e.g., a starting location, the current location of the robot  120  after finishing the previous journey) to a second location (e.g., a target destination). The robot  120  may receive inputs such as user commands to perform certain actions (e.g., scanning of inventory, moving an item, etc.) at certain locations. The planner  250  may include two types of routes, which corresponds to a spot check and a range scan. In a spot check, the planner  250  may receive an input that includes coordinates of one or more specific target locations. In response, the planner  250  plans a path for the robot  120  to travel to the target locations to perform an action. In a range scan, the input may include a range of coordinates corresponding to a range of target locations. In response, the planner  250  plans a path for the robot  120  to perform a full scan or actions for the range of target locations. 
     The planner  250  may plan the route of the robot  120  based on data provided by the visual reference engine  240  and the data provided by the state estimator  235 . For example, the visual reference engine  240  estimates the current location of the robot  120  by tracking the number of regularly shaped structures in the storage site  110  passed by the robot  120 . Based on the location information provided by the visual reference engine  240 , the planner  250  determines the route of the robot  120  and may adjust the movement of the robot  120  as the robot  120  travels along the route. 
     The planner  250  may also include a fail-safe mechanism in the case where the movement of the robot  120  has deviated from the plan. For example, if the planner  250  determines that the robot  120  has passed a target aisle and traveled too far away from the target aisle, the planner  250  may send signals to the FCU  225  to try to remedy the path. If the error is not remedied after a timeout or within a reasonable distance, or the planner  250  is unable to correctly determine the current location, the planner  250  may direct the FCU to land or to stop the robot  120 . 
     Relying on various location information, the planner  250  may also include algorithms for collision avoidance purposes. In some embodiments, the planner  250  relies on the distance information, the yaw angle, and center offset information relative to nearby objects to plan the movement of the robot  120  to provide sufficient clearance between the robot  120  and nearby objects. Alternatively, or additionally, the robot  120  may include one or more depth cameras such as a 360-degree depth camera set that generates distance data between the robot  120  and nearby objects. The planner  250  uses the location information from the depth cameras to perform collision avoidance. 
     The communication engine  255  and the I/O interface  260  are communication components to allow the robot  120  to communicate with other components in the system environment  100 . A robot  120  may use different communication protocols, wireless or wired, to communicate with an external component such as the base station  130 . Example communication protocols may include Wi-Fi, Bluetooth, NFC, USB, etc. that couple the robot  120  to the base station  130 . The robot  120  may transmit various types of data, such as image data, flight logs, location data, inventory data, and robot status information. The robot  120  may also receive inputs from an external source to specify the actions that need to be performed by the robot  120 . The commands may be automatically generated or manually generated by an administrator. The communication engine  255  may include algorithms for various communication protocols and standards, encoding, decoding, multiplexing, traffic control, data encryption, etc. for various communication processes. The I/O interface  260  may include software and hardware component such as hardware interface, antenna, and so forth for communication. 
     The robot  120  also includes a power source  265  used to power various components and the movement of the robot  120 . The power source  265  may be one or more batteries or a fuel tank. Example batteries may include lithium-ion batteries, lithium polymer (LiPo) batteries, fuel cells, and other suitable battery types. The batteries may be placed inside permanently or may be easily replaced. For example, batteries may be detachable so that the batteries may be swapped when the robot  120  returns to the base station  130 . 
     While  FIG.  2    illustrates various example components, a robot  120  may include additional components. For example, some mechanical features and components of the robot  120  are not shown in  FIG.  2   . Depending on its type, the robot  120  may include various types of motors, actuators, robotic arms, lifts, other movable components, other sensors for performing various tasks. 
     Continuing to refer to  FIG.  2   , an example base station  130  includes a processor  270 , a memory  275 , an I/O interface  280 , and a repowering unit  285 . In various embodiments, the base station  130  may include different, fewer, and/or additional components. 
     The base station  130  includes one or more processors  270  and one or more memories  275  that include one or more set of instructions for causing the processors  270  to carry out various processes that are implemented as one or more software modules. The base station  130  may provide inputs and commands to the robot  120  for performing various inventory management tasks. The base station  130  may also include an instruction set for performing swarm control among multiple robots  120 . Swarm control may include task allocation, routing and planning, coordination of movements among the robots to avoid collisions, etc. The base station  130  may serve as a central control unit to coordinate the robots  120 . The memory  275  may also include various sets of instructions for performing analysis of data and images downloaded from a robot  120 . The base station  130  may provide various degrees of data processing from raw data format conversion to full data processing that generates useful information for inventory management. Alternatively, or additionally, the base station  130  may directly upload the data downloaded from the robot  120  to a data store, such as the data store  160 . The base station  130  may also provide operation, administration, and management commands to the robot  120 . In some embodiments, the base station  130  can be controlled remotely by the user device  170 , the computing server  150 , or the inventory management system  140 . 
     The base station  130  may also include various types of I/O interfaces  280  for communications with the robot  120  and to the Internet. The base station  130  may communicate with the robot  120  continuously using a wireless protocol such as Wi-Fi or Bluetooth. In some embodiments, one or more components of the robot  120  in  FIG.  2    may be located in the base station and the base station may provide commands to the robot  120  for movement and navigation. Alternatively, or additionally, the base station  130  may also communicate with the robot  120  via short-range communication protocols such as NFC or wired connections when the robot  120  lands or stops at the base station  130 . The base station  130  may be connected to the network  180  such as the Internet. The wireless network (e.g., LAN) in some storage sites  110  may not have sufficient coverage. The base station  130  may be connected to the network  180  via an Ethernet cable. 
     The repowering unit  285  includes components that are used to detect the power level of the robot  120  and to repower the robot  120 . Repowering may be done by swapping the batteries, recharging the batteries, re-filling the fuel tank, etc. In some embodiments, the base station  130  includes mechanical actuators such as robotic arms to swap the batteries on the robot  120 . In another embodiment, the base station  130  may serve as the charging station for the robot  120  through wired charging or inductive charging. For example, the base station  130  may include a landing or resting pad that has an inductive coil underneath for wirelessly charging the robot  120  through the inductive coil in the robot. Other suitable ways to repower the robot  120  are also possible. 
     Example Inventory Management Process 
       FIG.  3    is a flowchart that depicts an example process for managing the inventory of a storage site, in accordance with some embodiments. The process may be implemented by a computer, which may be a single operation unit in a conventional sense (e.g., a single personal computer) or may be a set of distributed computing devices that cooperate to execute a set of instructions (e.g., a virtual machine, a distributed computing system, cloud computing, etc.). Also, while the computer is described in a singular form, the computer that performs the process in  FIG.  3    may include more than one computer that is associated with the computing server  150 , the inventory management system  140 , the robot  120 , the base station  130 , or the user device  170 . 
     In accordance with some embodiments, the computer receives  310  a configuration of a storage site  110 . The storage site  110  may be a warehouse, a retail store, or another suitable site. The configuration information of the storage site  110  may be uploaded to the robot  120  for the robot to navigate through the storage site  110 . The configuration information may include a total number of the regularly shaped structures in the storage site  110  and dimension information of the regularly shaped structures. The configuration information provided may take the form of a computer-aided design (CAD) drawing or another type of file format. The configuration may include the layout of the storage site  110 , such as the rack layout and placement of other regularly shaped structures. The layout may be a 2-dimensional layout. The computer extracts the number of sections, aisles, and racks and the number of rows and columns for each rack from the CAD drawing by counting those numbers as appeared in the CAD drawing. The computer may also extract the height and the width of the cells of the racks from the CAD drawing or from another source. In some embodiments, the computer does not need to extract the accurate distances between a given pair of racks, the width of each aisle, or the total length of the racks. Instead, the robot  120  may measure dimensions of aisles, racks, and cells from a depth sensor data or may use a counting method performed by the planner  250  in conjunction with the visual reference engine  240  to navigate through the storage site  110  by counting the number of rows and columns the robot  120  has passed. Hence, in some embodiments, the accurate dimensions of the racks may not be needed. 
     Some configuration information may also be manually inputted by an administrator of the storage site  110 . For example, the administrator may provide the number of sections, the number of aisles and racks in each section, and the size of the cells of the racks. The administrator may also input the number of rows and columns of each rack. 
     Alternatively, or additionally, the configuration information may also be obtained through a mapping process such as a pre-flight mapping or a mapping process that is conducted as the robot  120  carries out an inventory management task. For example, for a storage site  110  that newly implements the automated management process, an administrator may provide the size of the navigable space of the storage site for one or more mapping robots to count the numbers of sections, aisles, rows and columns of the regularly shaped structures in the storage site  110 . Again, in some embodiments, the mapping or the configuration information does not need to measure the accurate distance among racks or other structures in the storage site  110 . Instead, a robot  120  may navigate through the storage site  110  with only a rough layout of the storage site  110  by counting the regularly shaped structures along the path in order to identify a target location. The robotic system may gradually perform mapping or estimation of scales of various structures and locations as the robot  120  continues to perform various inventory management tasks. 
     The computer receives  320  inventory management data for inventory management operations at the storage site  110 . Certain inventory management data may be manually inputted by an administrator while other data may be downloaded from the inventory management system  140 . The inventory management data may include scheduling and planning for inventory management operations, including the frequency of the operations, time window, etc. For example, the management data may specify that each location of the racks in the storage site  110  is to be scanned every predetermined period (e.g., every day) and the inventory scanning process is to be performed in the evening by the robot  120  after the storage site is closed. The data in the inventory management system  140  may provide the barcodes and labels of items, the correct coordinates of the inventory, information regarding racks and other storage spaces that need to be vacant for incoming inventory, etc. The inventory management data may also include items that need to be retrieved from the storage site  110  (e.g., items on purchase orders that need to be shipped) for each day so that the robot  120  may need to focus on those items. 
     The computer generates  330  a plan for performing inventory management. For example, the computer may generate an automatic plan that includes various commands to direct the robot  120  to perform various scans. The commands may specify a range of locations that the robot  120  needs to scan or one or more specific locations that the robot  120  needs to go. The computer may estimate the time for each scanning trip and design the plan for each operation interval based on the available time for the robotic inventory management. For example, in certain storage sites  110 , robotic inventory management is not performed during the business hours. 
     The computer generates  340  various commands to operate one or more robots  120  to navigate the storage site  110  according to the plan and the information derived from the configuration of the storage site  110 . The robot  120  may navigate the storage site  110  by at least visually recognizing the regularly shaped structures in the storage sites and counting the number of regularly shaped structures. In some embodiments, in addition to the localization techniques such as VIO used, the robot  120  counts the number of racks, the number of rows, and the number of columns that it has passed to determine its current location along a path from a starting location to a target location without knowing the accurate distance and direction that it has traveled. 
     The scanning of inventory or other inventory management tasks may be performed autonomously by the robot  120 . In some embodiments, a scanning task begins at a base station at which the robot  120  receives  342  an input that includes coordinates of target locations in the storage site  110  or a range of target locations. The robot  120  departs  344  from the base station  130 . The robot  120  navigates  346  through the storage site  110  by visually recognizing regularly shaped structures. For example, the robot  120  tracks the number of regularly shaped structures that are passed by the robot  120 . The robot  120  makes turns and translation movements based on the recognized regularly shaped structures captured by the robot&#39;s image sensor  210 . Upon reaching the target location, the robot  120  may align itself with a reference point (e.g., the center location) of the target location. At the target location, the robot  120  captures  348  data (e.g., measurements, pictures, etc.) of the target location that may include the inventory item, barcodes, and labels on the boxes of the inventory item. If the initial command before the departure of the robot  120  includes multiple target locations or a range of target locations, the robot  120  continues to the next target locations by moving up, down, or sideways to the next location to continue to scanning operation. 
     Upon completion of a scanning trip, the robot  120  returns  350  to the base station  130  by counting the number of regularly shaped structures that the robot  120  has passed, in a reversed direction. The robot  120  may potentially recognize the structures that the robot has passed when the robot  120  travels to the target location. Alternatively, the robot  120  may also return to the base station  130  by reversing the path without any count. The base station  130  repowers the robot  120 . For example, the base station  130  provides the next commands for the robot  120  and swaps  352  the battery of the robot  120  so that the robot  120  can quickly return to service for another scanning trip. The used batteries may be charged at the base station  130 . The base station  130  also may download the data and images captured by the robot  120  and upload the data and images to the data store  160  for further process. Alternatively, the robot  120  may include a wireless communication component to send its data and images to the base station  130  or directly to the network  180 . 
     The computer performs  360  analyses of the data and images captured by the robot  120 . For example, the computer may compare the barcodes (including serial numbers) in the images captured by the robot  120  to the data stored in the inventory management system  140  to identify if any items are misplaced or missing in the storage site  110 . The computer may also determine other conditions of the inventory. The computer may generate a report to display at the user interface  175  for the administrator to take remedial actions for misplaced or missing inventory. For example, the report may be generated daily for the personnel in the storage site  110  to manually locate and move the misplaced items. Alternatively, or additionally, the computer may generate an automated plan for the robot  120  to move the misplaced inventory. The data and images captured by the robot  120  may also be used to confirm the removal or arrival of inventory items. 
     Example Navigation Process 
       FIG.  4    is a conceptual diagram of an example layout of a storage site  110  that is equipped with a robot  120 , in accordance with some embodiments.  FIG.  4    shows a two-dimensional layout of storage site  110  with an enlarged view of an example rack that is shown in inset  405 . The storage site  110  may be divided into different regions based on the regularly shaped structures. In this example, the regularly shaped structures are racks  410 . The storage site  110  may be divided into sections  415 , aisles  420 , rows  430  and columns  440 . For example, a section  415  is a group of racks. Each aisle may have two sides of racks. Each rack  410  may include one or more columns  440  and multiple rows  430 . The storage unit of a rack  410  may be referred to as a cell  450 . Each cell  450  may carry one or more pallets  460 . In this particular example, two pallets  460  are placed on each cell  450 . Inventory of the storage site  110  is carried on the pallets  460 . The divisions and nomenclature illustrated in  FIG.  4    are used as examples only. A storage site  110  in another embodiment may be divided in a different manner. 
     Each inventory item in the storage site  110  may be located on a pallet  460 . The target location (e.g., a pallet location) of the inventory item may be identified using a coordinate system. For example, an item placed on a pallet  460  may have an aisle number (A), a rack number (K), a row number (R), and a column number (C). For example, a pallet location coordinate of [A3, K1, R4, and C5] means that the pallet  460  is located at a rack  410  in the third aisle and the north rack. The location of the pallet  460  in the rack  410  is in the fourth row (counting from the ground) and the fifth column. In some cases, such as the particular layout shown in  FIG.  4   , an aisle  420  may include racks  410  on both sides. Additional coordinate information may be used to distinguish the racks  410  at the north side and the racks  410  at the south side of an aisle  420 . Alternatively, the top and bottom sides of the racks can have different aisle numbers. For a spot check, a robot  120  may be provided with a single coordinate if only one spot is provided or multiple coordinates if more than one spot is provided. For a range scan that checks a range of pallets  460 , the robot  120  may be provided with a range of coordinates, such as an aisle number, a rack number, a starting row, a starting column, an ending row, and an ending column. In some embodiments, the coordinate of a pallet location may also be referred to in a different manner. For example, in one case, the coordinate system may take the form of “aisle-rack-shelf-position.” The shelf number may correspond to the row number and the position number may correspond to the column number. 
     Referring to  FIG.  5    in conjunction with  FIG.  4   ,  FIG.  5    is a flowchart depicting an example navigation process of a robot  120 , in accordance with some embodiments. The robot  120  receives  510  a target location  474  of a storage site  110 . The target location  474  may be expressed in the coordinate system as discussed above in association with  FIG.  4   . The target location  474  may be received as an input command from a base station  130 . The input command may also include the action that the robot  120  needs to take, such as taking a picture at the target location  474  to capture the barcodes and labels of inventory items. The robot  120  may rely on the VIO unit  236  and the height estimator  238  to generate localization information. In one case, the starting location of a route is the base station  130 . In some cases, the starting location of a route may be any location at the storage site  110 . For example, the robot  120  may have recently completed a task and started another task without returning to the base station  130 . 
     The processors of the robot  120 , such as the one executing the planner  250 , control  520  the robot  120  to the target location  474  along a path  470 . The path  470  may be determined based on the coordinate of the target location  474 . The robot  120  may turn so that the image sensor  210  is facing the regularly shaped structures (e.g., the racks). The movement of the robot  120  to the target location  474  may include traveling to a certain aisle, taking a turn to enter the aisle, traveling horizontally to the target column, traveling vertically to the target row, and turning to the right angle facing the target location  474  to capture a picture of inventory items on the pallet  460 . 
     As the robot  120  moves to the target location  474 , the robot  120  captures  530  images of the storage site  110  using the image sensor  210 . The images captured may be in a sequence of images. The robot  120  receives the images captured by the image sensor  210  as the robot  120  moves along the path  470 . The images may capture the objects in the environment, including the regularly shaped structures such as the racks. For example, the robot  120  may use the algorithms in the visual reference engine  240  to visually recognize the regularly shaped structures. 
     The robot  120  analyzes  540  the images captured by the image sensor  210  to determine the current location of the robot  120  in the path  470  by tracking the number of regularly shaped structures in the storage site passed by the robot  120 . The robot  120  may use various image processing and object recognition techniques to identify the regularly shaped structures and to track the number of structures that the robot  120  has passed. Referring to the path  470  shown in  FIG.  4   , the robot  120 , facing the racks  410 , may travel to the turning point  476 . The robot  120  determines that it has passed two racks  410  so it has arrived at the target aisle. In response, the robot  120  turns counter-clockwise and enter the target aisle facing the target rack. The robot  120  counts the number of columns that it has passed until the robot  120  arrives at the target column. Depending on the target row, the robot  120  may travel vertically up or down to reach the target location. Upon reaching the target location, the robot  120  performs the action specified by the input command, such as taking a picture of the inventory at the target location. 
     Example Thin-Object Detection Process 
       FIG.  6    is a conceptual diagram illustrating a robot  600  detecting a thin object, in accordance with some embodiments. The robot  600  may be an aerial robot and may be an example of the robot  120  that is used in a storage site  110 . While  FIG.  1    through  FIG.  5    focuses on robots that navigate through a storage site, the robot  600  may also be used in other settings such as an outdoor environment or an urban environment in which thin objects are common. For example, a drone that is designed for city use may be equipped with thin-object detection capability to avoid collision with electrical wires, signs, and other thin objects. The robot  600  may include some or all of the components shown in  FIG.  2   . For example, in some embodiments, the robot  600  includes at least image sensor  210 , processor  215 , memory  220 , state estimator  235 , and IMU  230 . Other suitable configurations of the robot  600  are also possible. 
     Thin objects may refer to objects that have one or more dimensions that are thin. Examples of thin objects may include electrical wires, horizontal and vertical bars, other cables, etc. In the storage site  110 , thin objects such as wires, chains, and connection cables may be present. Various thin objects are present in different settings. From the perspective of the robot  600 , whether an object is thin may depend on how far away the object is relative to the robot  600 . For example, a first object that is farther away from the robot  600  will appear to be thinner than a second object that is closer to the robot  600 . In some embodiments, whether an object is thin may be defined based on the number of pixels occupied by the object in the thin dimension as the object is captured in an image. For example, in some embodiments, an object may be classified as thin when the thickness of the object in the thin dimension is fewer than 10 pixels thick. In some embodiments, an object may be classified as thin when the thickness of the object in the thin dimension is fewer than 5 pixels thick. In some embodiments, whether an object is thin may also be defined based on the angular resolution of an image sensor. For example, a typical camera installed in an aerial robot may have a certain angular resolution. An object may be classified as a thin object if the angle occupied by the object is lower than 2 degrees, 1.5 degrees, 1 degree, etc., depending on embodiments. In  FIG.  6   , a thin object  610  is represented by a horizontal bar  610  that is supported by its stand  612 . The setting in  FIG.  6    is only for illustration. Also, while the thin object  610  is shown as a horizontal thin object, a thin object may also be oriented differently, such as vertically or diagonally. 
     The robot  600  is equipped with one or more image sensors for capturing images in the environment. An image sensor used by the robot  600  may be a mono-camera with a single lens or a stereo camera with multiple lenses. In some embodiments, the thin-object detection technique enables a robot  600  equipped with a mono-camera to detect various thin objects. The robot  600  may use the image sensor to continuously capture images. For example, at a first location  620 , the image sensor captures a first image  630 . For simplicity, only the thin object  610  is shown in the first image  630 , although the first image  630  may include various things captured by the image sensor. 
     The robot  600  may include a thin object detector for identifying thin objects in the images captured by the image sensor. The thin object detector may be a software algorithm that is stored in a memory, such as memory  220 , and may be executed by one or more processors, such as the processor  215 , to analyze the images. The thin object detector may include one or more machine learning models used to analyze the images. The structure and training of the machine learning models are illustrated in  FIG.  8 A  and  FIG.  8 B . For example,  FIG.  8 B  illustrates an example structure of a convolutional neural network that may be specifically trained to identify thin objects in an image. The thin object detector may use the machine learning model to mark the pixels of the image that are identified as corresponding to the thin objects. The thin object detector may segment the thin objects from the rest of the scene. 
     The robot  600  may also include a state estimator that is used to determine the pose of the robot  600  relative to the thin object  610 . The pose of the robot  600  may be represented by the pose of the image sensor. A state estimator may be a combination of hardware sensors and software algorithms that are used to track the location and localization information of the robot  600 . An example of the state estimator may be the state estimator  235  discussed in  FIG.  2   . The state estimator may also include an IMU  230  that generates acceleration and orientation data. The acceleration data may be converted to distance data. The robot  600  uses the data from the IMU  230  and potentially also the VIO  236  to determine the pose of the robot  600  relative to the thin object  610 . 
     The robot  600  includes a depth estimator that estimates the distance between the robot  600  and the thin object  610  at a given position. The depth estimator may be a software algorithm that is stored in a memory and executed by one or more processors to use results from the thin object detector and the state estimator to determine an estimated distance between the robot  600  and the thin object  610 . The depth estimator determines the estimated distance based on two or more images that are captured at different locations by comparing the pixels of the thin object  610  captured in the images. The robot  600  captures the thin object  610  in different locations of the images as the robot  600  moves from a first location to a second location. The depth estimator compares the locations of the pixels and the IMU distance data to estimate the distance between the robot  600  and the thin object  610 . 
     By way of example, at the first position  620 , the thin object detector marks the pixels corresponding to the thin object  610  in the first image  630 . The depth estimator receives the thin object pixel locations in the first image  630 . The depth estimator identifies pixel locations  632  of one or more particular features of the thin object  610 . A feature may be a readily identifiable part or region of the thin object  610  that is projected to be continuously identifiable as the image sensor captures the thin object  610  from different perspectives. For example, features can be the centroid, the center, the far ends, quantiles, identifiable subparts, or any other suitable parts that are associated with the thin object  610 . In the first image  630 , the depth estimator identifies both ends of the thin object  610  as the features and marks the two pixel locations  632  corresponding to the far ends. For example, the depth estimator may identify, based on the coordinates of the pixels, pixel locations with the highest and lowest values in a particular dimension to determine the two far ends. 
     The depth estimator compares the pixel locations of the features after the robot  600  travels from the first location  620  to a second location  640 . At the second location  640 , the image sensor of the robot  600  captures a second image  650 . In the second image  650 , the thin object  610  appears to be longer because the robot  600  is located closer to the thin object  610  at the second location  640 . The robot  600  may have moved vertically so that the level of the thin object  610  may also be changed. The thin object detector analyzes the second image  650  and marks the pixels corresponding to the thin object  610 . The depth estimator receives the pixel locations from the object detector and identifies the features of the thin object  610 . For example, the depth estimator again identifies the two far ends of the thin object  610  and determines the pixel locations  652  corresponding to the two far ends. The pixel locations  652  are changed relative to the pixel locations  632  as the robot  600  moves to the second location  640 . 
     The depth estimator determines the estimated distance between the robot  600  and the thin object  610  based on the differences between the pixel locations  632  and  652  and based on the distance data and poses provided by the state estimator. For example, the distance data generated by an IMU may be used to estimate the distance traveled by the robot  600  from the first location  620  to the second location  640 . The state estimator may also provide the poses of the robot  600  in the two locations. The distance between the robot  600  and the thin object  610  may be calculated based on mathematical operations such as projection, linear transformation, and geometric relationships. 
       FIG.  7    is a flowchart depicting an example process of a robot for detecting thin objects in the environment, in accordance with some embodiments. The process illustrated in  FIG.  7    may be a detailed example of the process illustrated in  FIG.  6   . The process illustrated in  FIG.  7    may be executed through a software algorithm that is stored as computer instructions that are executable by one or more processors (e.g., CPU) of a robot. The instructions, when executed by the processors, cause the processors to perform various steps described in the process. In various embodiments, one or more steps in the process may be skipped or be changed. The robot that performs the process may be the robot  600 . 
     The robot receives  710  a first image of an environment. The first image may be captured at a first location. The robot is equipped with an image sensor that captures the image. In some embodiments, the image may be capture by a mono-camera and may not need to include point cloud or any three-dimensional data. The first location may be the instant location of the robot as the robot travels along a path. The process illustrated in  FIG.  7    may be a continuous process that may be repeatedly performed as the robot travels to different locations. The image may capture various objects in the environment. Some of the objects may be thin objects that are difficult to be detected using conventional image sensors or point cloud data because the objects may correspond to a very small number of pixels in one or more dimensions. A robot is often unable to generate conventional point cloud data with depth data for thin objects because the changes of depth in the thin object locations are often indistinguishable from noise. 
     The robot identifies  720  an object in the first image. The object may be a thin object. The robot may include a thin object detector that executes a machine learning model trained to identify certain targeted objects. For example, the machine learning model may be a convolutional neural network (CNN) that includes one or more dilated convolutional layers. An example of the structure of CNN is shown in  FIG.  8 B . The machine learning model receives the first image as an input and tags the pixels corresponding to the identified object as the outputs. For example, the CNN may be specifically trained to identify thin objects. For example, the CNN may be trained using a training set of images of various environments that include different targeted objects. The training of the CNN may include iteratively reducing the errors in the identification of thin object locations to the training set of images. Detailed procedure in training a machine learning model is discussed in  FIG.  8 A . 
     Although identifying thin objects is discussed as an example, the process depicted in  FIG.  7    may also be used to identify other types of objects and determine the distance between the robot and the objects. 
     The robot identifies  730  one or more first pixels in the first image. The one or more first pixels may correspond to one or more targeted features of the identified object. The identification of the first pixels may include various sub-steps. By way of example, the robot may tag pixels in the first image projected to correspond to the identified object. The robot may cluster the pixels to form a plurality of contours. The robot may merge the contours to form a merged contour. The robot may identify targeted features from pixels in the merged contour. 
     The process may involve various image segmentation techniques and object identification techniques that separate pixels corresponding to the identified objects from the background. The image segmentation may be carried out by any suitable algorithms. In some embodiments, the robot the machine learning model to perform the image segmentation and to assign tags to the pixels. In some embodiments, the robot may also input a series of images to the machine learning model, which may output image segmentations from the series of images by taking into account the object appearing continuously in the series of images. Alternative to or in addition to the CNN, other types of machine learning models, such as another type of a neural network, clustering, Markov random field (MRF), etc., may also be used in the image segmentation process. Alternative to or in addition to using any machine learning techniques, other image segmentation algorithms such as edge detection algorithms (e.g., Canny operator, Laplacian operator, Sobel operator, Prewitt operator), corner detection algorithms, Hough transform, and other suitable feature detection algorithms may also be used. 
     From the pixels that are tagged with symbols representing the identified objects, the robot may create contours of regions of those pixels. Each contour may be referred to as a cluster of pixels. Clustering of those pixels may be based on distances among the pixels, the colors of the pixels, the intensities of the pixels, and/or other suitable characteristics of the pixels. The robot may cluster similar nearby pixels (e.g., in terms of distances, colors, and/or intensities) to create a contour that is likely to correspond to a region of the identified object. Multiple clusters that correspond to various sub-regions may be created for each identified object (e.g., each wire or another type of thin object). The robot may determine a reference point for each contour. The reference point may be the centroid, the most extreme point in one direction, or any relevant reference point. For example, the robot may determine the average of the pixel locations for pixels that are within the contour to determine the centroid. 
     The robot may perform noise filtering and contour merging that merges contours based on their respective reference points. For noise filtering, the robot may filter the contours based on sizes before merging. For example, the robot may regard contours whose areas are smaller than a threshold (e.g., contours that are too small) as noise. The robot keeps contours that are sufficiently large. For contour merging, in some cases, the clustering algorithm results in pixels corresponding to the identified object being classified into multiple contours (e.g., multiple regions of a wire). The robot merges contours that likely represent the same identified object. The merging may be based on the positions of the reference points of the contours and the boundaries of the contours. For example, for a horizontal wire, the robot may identify contours that have reference points in a similar vertical level and merge those contours. In some cases when two contours are merged, pixels between the two contours, which may belong to a smaller cluster or may not be identified in any cluster, may also be classified as the same structure. Merging may be based on the distance between two reference points of two contours. If the distance is smaller than a threshold level, the robot may merge the two contours. 
     Upon separating the pixels corresponding to the identified object from the rest of the pixels, the robot identifies one or more targeted features in the objects. The targeted features may be specific locations of the objects relative to the total length of the object. For example, in the case of thin objects, the far ends, centers, quantiles may be sued as targeted features. For other types of objects, in addition to or alternative to specific locations, the targeted features may also be identifiable parts of the objects. For example, in the setting of a storage site and where structures are the identified objects, the structures may include identifiable parts such as corner locations, screw locations, pattern locations, sign locations, etc. The robot may identify those parts using object recognition techniques. In the case of the targeted features being specific locations relative to the total length of the object, the robot may use the coordinate values of the segmented pixels to determine the targeted feature locations. For example, in some embodiments, the robot tracks the two far ends of an identified thin object and stores the coordinates for the first image. 
     The robot receives  740  a second image of the scene. The second image is captured at a second location that is different from the first location. As the robot travels, its image sensor continues to capture additional images of the scene. In some cases, the second image may be the immediately succeeding frame of the first image. In other cases, the second image is a succeeding frame that shows a sufficiently significant change in the content of the image compared to the first image. While  FIG.  7    is illustrated with two images, in some embodiments, the robot may also analyze a series of multiple images to increase the accuracy of the estimation of the distance between the robot and the identified object. Also, the process may be repeatedly performed to dynamically determine the distance as the robot travels to different locations. 
     The robot receives  750  distance data of the robot. The distance data estimates the movement of the aerial robot from the first location to the second location. The distance data may be generated by an IMU. The robot uses a state estimator to determine the distance traveled by the robot from the first location to the second location. The state estimator may also estimate the pose of the image sensor relative to the identified object respectively in the first location and the second location. 
     The robot identifies  760  one or more second pixels in the second image. The one or more second pixels correspond to the targeted features of the object. The identification of the second pixels is similar to the identification of the first pixels discussed in step  730 . In some embodiments, the robot also relies on the distance data and pose data generated by the state estimator to identify the second pixels. Based on the results of the state estimator, the robot may estimate the distance traveled and the change of orientation of the robot. The robot may project the proximity of the targeted features (locations of the one or more first pixels) in the second image based on the distance data and the pose data. The robot may use the projection to identify the targeted features. For example, the robot may search for the targeted features in the projected proximity. The result may be used in conjunction with the object recognition result generated by the CNN. 
     The robot determines  770  an estimated distance between the robot and the object based on changes of locations of the one or more second pixels from the one or more first pixels relative to the movement of the robot provided by the distance data. The estimation may be determined by the depth estimator of the robot. The distance between the robot and the identified object may be calculated based on mathematical operations such as projection, linear transformation, and geometric relationships. For example, if the movement of the robot is b between the first pixels and the second pixels and the disparity of corresponding pixel location is d, the distance z between the robot and the object can be computed by z=f*b/d, where f is the focal length of the camera. The depth estimator may send the distance to a flight control unit to direct the robot to avoid any collision with the identified object. For example, the flight control unit may change the route of the robot to avoid the object. 
     Example Machine Learning Models 
     In various embodiments, a wide variety of machine learning techniques may be used. Examples include different forms of supervised learning, unsupervised learning, and semi-supervised learning such as decision trees, support vector machines (SVMs), regression, Bayesian networks, and genetic algorithms. Deep learning techniques such as neural networks, including convolutional neural networks (CNN), recurrent neural networks (RNN) and long short-term memory networks (LSTM), may also be used. For example, various object recognitions performed by visual reference engine  240 , localization, recognition of objects and particularly thin objects, and other processes may apply one or more machine learning and deep learning techniques. 
     In various embodiments, the training techniques for a machine learning model may be supervised, semi-supervised, or unsupervised. In supervised learning, the machine learning models may be trained with a set of training samples that are labeled. For example, for a machine learning model trained to classify objects, the training samples may be different pictures of objects labeled with the type of objects. The labels for each training sample may be binary or multi-class. In training a machine learning model for image segmentation, the training samples may be pictures of regularly shaped objects in various storage sites with segments of the images manually identified. In some cases, an unsupervised learning technique may be used. The samples used in training are not labeled. Various unsupervised learning technique such as clustering may be used. In some cases, the training may be semi-supervised with training set having a mix of labeled samples and unlabeled samples. 
     A machine learning model may be associated with an objective function, which generates a metric value that describes the objective goal of the training process. For example, the training may intend to reduce the error rate of the model in generating predictions. In such a case, the objective function may monitor the error rate of the machine learning model. In object recognition (e.g., object detection and classification), the objective function of the machine learning algorithm may be the training error rate in classifying objects in a training set. Such an objective function may be called a loss function. Other forms of objective functions may also be used, particularly for unsupervised learning models whose error rates are not easily determined due to the lack of labels. In image segmentation, the objective function may correspond to the difference between the model&#39;s predicted segments and the manually identified segments in the training sets. In various embodiments, the error rate may be measured as cross-entropy loss, L1 loss (e.g., the sum of absolute differences between the predicted values and the actual value), L2 loss (e.g., the sum of squared distances). 
     Referring to  FIG.  8 A , a structure of an example CNN is illustrated, in accordance with some embodiments. The CNN  800  may receive an input  810  and generate an output  820 . The CNN  800  may include different kinds of layers, such as convolutional layers  830 , pooling layers  840 , recurrent layers  850 , full connected layers  860 , and custom layers  870 . A convolutional layer  830  convolves the input of the layer (e.g., an image) with one or more kernels to generate different types of images that are filtered by the kernels to generate feature maps. Each convolution result may be associated with an activation function. A convolutional layer  830  may be followed by a pooling layer  840  that selects the maximum value (max pooling) or average value (average pooling) from the portion of the input covered by the kernel size. The pooling layer  840  reduces the spatial size of the extracted features. In some embodiments, a pair of convolutional layer  830  and pooling layer  840  may be followed by a recurrent layer  850  that includes one or more feedback loop  855 . The feedback  855  may be used to account for spatial relationships of the features in an image or temporal relationships of the objects in the image. The layers  830 ,  840 , and  850  may be followed in multiple fully connected layers  860  that have nodes (represented by squares in  FIG.  8 A ) connected to each other. The fully connected layers  860  may be used for classification and object detection. In some embodiments, one or more custom layers  870  may also be presented for the generation of a specific format of output  820 . For example, a custom layer may be used for image segmentation for labeling pixels of an image input with different segment labels. 
     The order of layers and the number of layers of the CNN  800  in  FIG.  8 A  is for example only. In various embodiments, a CNN  800  includes one or more convolutional layer  830  but may or may not include any pooling layer  840  or recurrent layer  850 . If a pooling layer  840  is present, not all convolutional layers  830  are always followed by a pooling layer  840 . A recurrent layer may also be positioned differently at other locations of the CNN. For each convolutional layer  830 , the sizes of kernels (e.g., 3×3, 5×5, 7×7, etc.) and the numbers of kernels allowed to be learned may be different from other convolutional layers  830 . 
     A machine learning model may include certain layers, nodes, kernels and/or coefficients. Training of a neural network, such as the CNN  800 , may include forward propagation and backpropagation. Each layer in a neural network may include one or more nodes, which may be fully or partially connected to other nodes in adjacent layers. In forward propagation, the neural network performs the computation in the forward direction based on outputs of a preceding layer. The operation of a node may be defined by one or more functions. The functions that define the operation of a node may include various computation operations such as convolution of data with one or more kernels, pooling, recurrent loop in RNN, various gates in LSTM, etc. The functions may also include an activation function that adjusts the weight of the output of the node. Nodes in different layers may be associated with different functions. 
     Each of the functions in the neural network may be associated with different coefficients (e.g. weights and kernel coefficients) that are adjustable during training. In addition, some of the nodes in a neural network may also be associated with an activation function that decides the weight of the output of the node in forward propagation. Common activation functions may include step functions, linear functions, sigmoid functions, hyperbolic tangent functions (tanh), and rectified linear unit functions (ReLU). After an input is provided into the neural network and passes through a neural network in the forward direction, the results may be compared to the training labels or other values in the training set to determine the neural network&#39;s performance. The process of prediction may be repeated for other images in the training sets to compute the value of the objective function in a particular training round. In turn, the neural network performs backpropagation by using gradient descent such as stochastic gradient descent (SGD) to adjust the coefficients in various functions to improve the value of the objective function. 
     Multiple rounds of forward propagation and backpropagation may be performed. Training may be completed when the objective function has become sufficiently stable (e.g., the machine learning model has converged) or after a predetermined number of rounds for a particular set of training samples. The trained machine learning model can be used for performing prediction, object detection, image segmentation, or another suitable task for which the model is trained. 
       FIG.  8 B  is a conceptual diagram illustrating an example CNN  880  that is structured and trained to identify thin objects, in accordance with some embodiments. The CNN  880  may be an example of the CNN  800  and the training techniques discussed in  FIG.  8 A  may also be used for CNN  880 . The CNN  880  may include front end layers  882  and context layers  884 . The front end layers  882  includes convolutional layers  830  and pooling layers  840  that are similar to the CNN  800 . The size of kernels, the number of convolutional layers  830 , and the pooling parameters may be customized, depending on embodiments. The front end layers may be used to detect edges in the input images and identify certain low-level patterns in the images. 
     The context layers  884  include one or more dilated convolutional layers  890  that are used to generate the output  895 . Each dilated convolution layer  890  may be associated with a dilation factor. A kernel with a dilation factor will be expanded in size and filled with zeros in the expanded space. For example, a dilation factor of 2 inserts a zero between two values in a row and inserts rows with zeros between the original rows. A higher dilation factor inserts more zero and further expands the size of the kernel. Common dilation factors may be 2, 4, 8, 16, etc. A dilated convolution may allow the CNN  880  to distinguish larger patterns from more localized patterns. The context layers  884  includes one or more dilated convolutional layers  890  that may improve the performance of the CNN  880  in detecting thin objects such as wires and cables. In some embodiments, some of the dilated convolutional layers  890  may have increasing dilation factors. For example, a series of dilated convolutional layers  890  may have dilation factors of d1-d2-d4-d8. 
     The CNN  880  may be trained with a set of training samples that are images including various thin objects. The training samples may be generated by actual images of various scenes, indoor, outdoosr, or in different environments with different backgrounds. To generate more training samples, some of the images may be further manipulated, such as by rotating, scaling, skewing, adjusting contrast, and adjusting the color tone of the images. The thin objects in the images may also be adjusted to make some of the images simulate various conditions such as different lighting, weather, etc. 
     Computing Machine Architecture 
       FIG.  9    is a block diagram illustrating components of an example computing machine that is capable of reading instructions from a computer-readable medium and execute them in a processor (or controller). A computer described herein may include a single computing machine shown in  FIG.  9   , a virtual machine, a distributed computing system that includes multiples nodes of computing machines shown in  FIG.  9   , or any other suitable arrangement of computing devices. 
     By way of example,  FIG.  9    shows a diagrammatic representation of a computing machine in the example form of a computer system  900  within which instructions  924  (e.g., software, program code, or machine code), which may be stored in a computer-readable medium for causing the machine to perform any one or more of the processes discussed herein may be executed. In some embodiments, the computing machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a network deployment, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. 
     The structure of a computing machine described in  FIG.  9    may correspond to any software, hardware, or combined components shown in  FIGS.  1  and  2   , including but not limited to, the inventory management system  140 , the computing server  150 , the data store  160 , the user device  170 , and various engines, modules, interfaces, terminals, and machines shown in  FIG.  2   . While  FIG.  9    shows various hardware and software elements, each of the components described in  FIGS.  1  and  2    may include additional or fewer elements. 
     By way of example, a computing machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smartphone, a web appliance, a network router, an internet of things (IoT) device, a switch or bridge, or any machine capable of executing instructions  924  that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute instructions  924  to perform any one or more of the methodologies discussed herein. 
     The example computer system  900  includes one or more processors (generally, processor  902 ) (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application-specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any combination of these), a main memory  904 , and a non-volatile memory  906 , which are configured to communicate with each other via a bus  908 . The computer system  900  may further include graphics display unit  910  (e.g., a plasma display panel (PDP), a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)). The computer system  900  may also include alphanumeric input device  912  (e.g., a keyboard), a cursor control device  914  (e.g., a mouse, a trackball, a joystick, a motion sensor, or other pointing instrument), a storage unit  916 , a signal generation device  918  (e.g., a speaker), and a network interface device  920 , which also are configured to communicate via the bus  908 . 
     The storage unit  916  includes a computer-readable medium  922  on which is stored instructions  924  embodying any one or more of the methodologies or functions described herein. The instructions  924  may also reside, completely or at least partially, within the main memory  904  or within the processor  902  (e.g., within a processor&#39;s cache memory) during execution thereof by the computer system  900 , the main memory  904  and the processor  902  also constituting computer-readable media. The instructions  924  may be transmitted or received over a network  926  via the network interface device  920 . 
     While computer-readable medium  922  is shown in an example embodiment to be a single medium, the term “computer-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) able to store instructions (e.g., instructions  924 ). The computer-readable medium may include any medium that is capable of storing instructions (e.g., instructions  924 ) for execution by the machine and that cause the machine to perform any one or more of the methodologies disclosed herein. The computer-readable medium may include, but not be limited to, data repositories in the form of solid-state memories, optical media, and magnetic media. The computer-readable medium does not include a transitory medium such as a signal or a carrier wave. 
     Additional Configuration Considerations 
     Certain embodiments are described herein as including logic or a number of components, engines, modules, or mechanisms. Engines may constitute either software modules (e.g., code embodied on a computer-readable medium) or hardware modules. A hardware engine is a tangible unit capable of performing certain operations and may be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., a standalone, client or server computer system) or one or more hardware engines of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware engine that operates to perform certain operations as described herein. 
     In various embodiments, a hardware engine may be implemented mechanically or electronically. For example, a hardware engine may comprise dedicated circuitry or logic that is permanently configured (e.g., as a special-purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware engine may also comprise programmable logic or circuitry (e.g., as encompassed within a general-purpose processor or another programmable processor) that is temporarily configured by software to perform certain operations. It will be appreciated that the decision to implement a hardware engine mechanically, in dedicated and permanently configured circuitry, or temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations. 
     The various operations of example methods described herein may be performed, at least partially, by one or more processors, e.g., processor  902 , that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented engines that operate to perform one or more operations or functions. The engines referred to herein may, in some example embodiments, comprise processor-implemented engines. 
     The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the one or more processors or processor-implemented modules may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the one or more processors or processor-implemented modules may be distributed across a number of geographic locations. 
     Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs for a similar system or process through the disclosed principles herein. Thus, while particular embodiments and applications have been illustrated and described, it is to be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, changes, and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.