Patent Description:
<CIT> discloses a computer-implemented method, in the form of a guidance system for unmanned aerial inspection in confined work environment.

<CIT> discloses a navigation system for UAVs which makes use of optical tags such as QR code which identify spatial positions. The UAVs are adapted to use said tags as reference positions in the case that GPS signals are weak or absent.

This specification describes systems and methods for autonomous aerial imaging and environmental sensing, e.g., humidity, temperature, noise, Wi-Fi spectrum, airflow, of a datacenter.

In general, one innovative aspect of the subject matter described in this specification may be embodied in methods that include the actions of executing, within a datacenter and by an unmanned aerial vehicle (UAV) in data communication with a server, a connectivity-enabled navigation process to collect data from the datacenter; determining, during execution of the connectivity-enabled navigation process and by the UAV, that connection to the server is lost so that the connectivity-enabled navigation process cannot be continued; and in response to determining that connection to the server is lost, executing, by the UAV, an autonomous navigation process until the connection to the server is recovered, wherein the autonomous navigation process navigates the datacenter using visual tags positioned in the datacenter.

Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination thereof installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus (e.g., one or more computers or computer processors), cause the apparatus to perform the actions.

The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. In some implementations executing the autonomous navigation process comprises: activating an onboard imaging system; and searching, using the onboard imaging system, for a visual tag positioned in the datacenter; and in response to successfully identifying, by the onboard imaging system, a visual tag, flying, by the UAV towards the visual tag.

In some implementations the method further comprises, in response to unsuccessfully identifying a visual tag, widening the search for the visual tag or returning to a docking station.

In some implementations the method further comprises, in response to successfully identifying, by the onboard imaging system, a visual tag, extracting metadata from the visual tag to determine properties of the visual tag, the properties comprising a location of the visual tag; determining, by the UAV and using the properties of the visual tag, whether to fly to the visual tag or not; and in response to determining to fly to the visual tag, flying to the visual tag; or in response to determining not to fly to the visual tag, searching for another visual tag.

In some implementations determining, by the UAV and using the properties of the visual tag, whether to fly to the visual tag or not comprises one or more of: determining whether the location of the visual tag is within a predetermined set of distance offsets from a known location from which to collect data or not and in response to determining that the location of the visual tag corresponds to a location from which to collect data, flying to the visual tag; determining whether data has already been collected from the location of the visual tag during a current data collection mission or not and in response to determining that data has not already been collected from the location of the visual tag, flying to the visual tag; or determining whether a name of the visual tag indicates that the location of the visual tag is relevant to the current data collection mission or not and in response to determining that the name of the visual tag indicates that the location of the visual tag is relevant to the current data collection mission, flying to the visual tag.

In some implementations the method further comprises one or more of: collecting data from a position that is within a predetermined set of distance offsets from the visual tag or collecting data whilst flying towards the visual tag.

In some implementations the method further comprises, after collecting data from a position that is within a predetermined set of distance offsets from the visual tag or collecting data whilst flying towards the visual tag, searching for another visual tag positioned in the datacenter.

In some implementations the method further comprises determining, by the UAV and using the properties of the visual tag, whether to collect a sensor package for collecting sensor data from a position that is within a predetermined set of distance offsets from the visual tag; in response to determining to collect a sensor package, collecting the sensor package and using the sensor package to collect sensor data from a position that is within a predetermined set of distance offsets from the visual tag; and returning the sensor package.

In some implementations searching for a visual tag positioned in the datacenter comprises querying an onboard processing system for a visual tag that is likely to be close to a current location of the UAV and flying towards the visual tag.

In some implementations the onboard imaging system is separate from an onboard imaging system used to collect image data from the datacenter.

In some implementations the visual tags comprise geotagged visual media and metadata represented by the visual tags comprises one or more of a physical tag location, a logical tag location, tag altitude, or tag name.

In some implementations the visual tags are positioned independent of pre-defined data collection mission or path and the visual tags are positioned on one or more of: a floor of the datacenter, walls of the datacenter, on racks at ends of datacenter rows, a ceiling of the datacenter, or a top surface of racks included in the datacenter.

In some implementations the UAV collects image data using an onboard imaging system and environmental data using one or more sensors, the environmental data comprising a measured temperature, humidity, noise, Wi-Fi spectrum, or wind speed.

In some implementations the method further comprises determining, during execution of the autonomous navigation process and by the UAV, that the connection to the server has returned so that the connectivity-enabled navigation process can be continued; and in response to determining that the connection to the server has returned, executing, by the UAV, the connectivity-enabled navigation process.

In some implementations executing, by the UAV, the connectivity-enabled navigation process comprises performing a reconciliation of the connectivity-enabled navigation process, the performing comprising: determining a location at which the connection to the server was lost and a location at which the connection to the server returned; determining whether waypoints or data collection steps were missed during the autonomous navigation process between the location at which the connection to the server was lost and the location at which the connection to the server returned; and in response to determining that waypoints or data collection steps were missed, adjusting the connectivity-enabled navigation process to include the missed waypoints or data collection steps.

In some implementations executing, by the UAV, the connectivity-enabled navigation process comprises performing a reconciliation of the connectivity-enabled navigation process, the performing comprising: determining a location at which the connection to the server was lost and a location at which the connection to the server returned; determining whether additional waypoints or data collection steps were performed during the autonomous navigation process; and in response to determining that additional waypoints or data collection steps were performed during the autonomous navigation process, adjusting the connectivity-enabled navigation process to remove the additional waypoints or data collection steps.

In some implementations the method further comprises storing, by the UAV, data collected during the autonomous navigation process; and uploading, by the UAV, the stored data to the server when the connection to the server is recovered.

In some implementations the autonomous navigation process further navigates the datacenter using path finding performed by an artificial intelligence layer included in the UAV.

Some implementations of the subject matter described herein may realize, in certain instances, one or more of the following advantages.

A system implementing the presently described autonomous datacenter aerial imaging and sensing techniques can monitor and collect data from a datacenter without requiring GPS signals or a constant connection to a navigation controller or server. In addition, the presently described autonomous datacenter aerial imaging and sensing techniques can be combined with a connectivity-based navigation process and executed as part of a recovery operation in the event that a connection to a navigation controller/server goes down. Compared to conventional recovery operations, e.g., hovering in place until connectivity is recovered, landing in place until connectivity is recovered, returning to a docking station, or backtracking a recent flightpath until connectivity is recovered or until it reaches its previous docking station, the presently described recovery operation are more efficient and less resource intensive.

In addition, a system implementing the presently described autonomous datacenter aerial imaging and sensing techniques can collect a richer stream of data, e.g., compared to conventional systems that do not use drones, since drones can be used to navigate to areas that a person or other vehicle might not otherwise be able to access. In addition, data collection can be completed much faster, e.g., compared to conventional systems that do not use drones. Since datacenters can be very large, this speed up can be particularly beneficial.

In addition, a system implementing the presently described autonomous datacenter aerial imaging and sensing techniques can perform imaging or environmental sensing at different heights, e.g. measuring temperature or Wi-Fi signal strength at multiple heights. This can be particularly beneficial since data centers can include hard to reach areas or infrastructure that is deployed high up, which would be difficult to image from floor levels. As an example, the angle of read from floor levels make some device barcodes impossible to read. Conventional solutions (different from the presently described techniques) to this problem require that the camera is placed on a mast that can be raised or lowered. However, this further slows down data acquisition and increases device costs.

In addition, a system implementing the presently described autonomous datacenter aerial imaging and sensing techniques uses aerial highways that do not interfere with existing personnel and material paths. Accordingly, the speed of reaching a target location is improved.

Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

This specification describes systems and methods for imaging and sensing a datacenter using an unmanned aerial vehicle (UAV) that performs an autonomous navigation process. Under the autonomous navigation process, the UAV navigates the datacenter using visual tags positioned in the datacenter and does not require connectivity to an operator or navigation system. Therefore, imaging and sensing of an area can be performed without connectivity and collected data can be reported when the UAV returns to an area with connectivity. <FIG> shows an example of an environment <NUM> for autonomous datacenter aerial imaging and sensing. An unmanned aerial vehicle (UAV) <NUM>, e.g., a drone, navigates a datacenter to collect data from predetermined target locations within the datacenter. The datacenter can include multiple datacenter rows, e.g., datacenter rows 104a and 104b, where each datacenter row includes multiple racks, e.g., racks 106a and 106b, that each house servers and other computing components such as networking devices or telecommunication devices. The datacenter can also include power equipment, water treatment facilities, and cooling systems that keep the servers operating smoothly.

The UAV <NUM> can include an imaging system and other sensors, e.g., thermal, humidity, noise, airflow, or Wi-Fi spectrum sensors, that collect data <NUM> from within the data center as part of an imaging and sensing mission. Example missions include new datacenter discovery, detailed scans or checks of a particular area of the datacenter or a particular rack/device in the datacenter, or datacenter audits. The collected data <NUM> can include image data or environmental data, e.g., data representing a measured humidity, temperature, noise level, airflow speed, or Wi-Fi spectrum. For example, the UAV <NUM> can include a first onboard imaging system that collects images of target locations within the data center, e.g., image <NUM>, of specific rows, racks, or servers. The UAV <NUM> can also include an onboard processor that stores the collected images in a data store and uploads the collected images to an external server <NUM> for analysis, e.g., in real time using wireless communication or when the UAV <NUM> docks at a docking station.

The UAV <NUM> can also include a second onboard imaging system that is used to navigate the datacenter. The second onboard imaging system can be separate to the first onboard imaging system used to collect image data. For example, the UAV <NUM> can use the second onboard imaging system to search the datacenter for visual tags, e.g., visual tag <NUM>, positioned in the datacenter. The visual tags can include geotagged visual media, e.g., geotagged photographs, images or QR codes. Metadata represented by the visual tags can include, e.g., a physical tag location, logical tag location, tag altitude, and tag name. In some implementations visual tags in a particular area can be commonly named, e.g., visual tags near datacenter rows can be named differently to visual tags near a water treatment facility.

The position of the visual tags in the datacenter can vary. As shown in example environment <NUM>, in some implementations the visual tags can be placed on the floor. In these implementations the visual tags could also be used by personnel or other imaging systems such as a vehicular imaging systems that operate in the datacenter. In other implementations the visual tags can be placed on walls in the datacenter, e.g., at the end of a row of racks, on the ceiling of the datacenter, or on the top of one or more racks, e.g., a top surface of a rack. This can be beneficial since such visual tags would not need regular maintenance and would be less likely to be covered by other objects in the datacenter (compared to, e.g., visual tags positioned on the floor which could easily get scratched or covered). In some implementations the visual tags can be placed at an optimal flying altitude, e.g., an altitude with a least amount of obstacles for the UAV to navigate. In some implementations the exact positioning of the visual tags can depend on the capabilities of the second onboard imaging system, e.g., the visual tags can be spaced far apart if the onboard imaging system has a high resolution. Generally, the visual tags need not be positioned on a pre-programmed path. The positioning can be independent of data collection missions.

Each visual tag can be associated with a physical or logical location in the datacenter. The UAV <NUM> can use these locations to identify its current location and navigate through the datacenter. For example, when the UAV <NUM> identifies a visual tag using the second onboard imaging system, the UAV <NUM> can determine whether to navigate towards the visual tag or not, e.g., whether the visual tag is a target location from which to collect data. If the visual tag is a target location from which to collect data, the UAV <NUM> can fly to the visual tag and collect the required data. If the visual tag is not a target location from which to collect data, the UAV <NUM> can use the second onboard imaging system to search for another, different visual tag. In this manner, the UAV <NUM> can hopscotch from one visual tag to another and collect all necessary data.

<FIG> is a flowchart of an example process <NUM> for collecting data from a datacenter using an autonomous navigation process. For convenience, the process <NUM> will be described as being performed by an UAV included in an autonomous datacenter aerial imaging and sensing system. For example, the unmanned aerial vehicle (UAV) <NUM> of <FIG>, appropriately programmed, can perform example process <NUM>.

The UAV uses its onboard imaging system to search for a visual tag (step <NUM>). In some implementations the UAV can use its most recent known location to query an onboard processing system for visual tags that are likely to be close to its current location. The UAV can then fly at a low, safe speed in a direction towards one of the close visual tags. If the UAV cannot find any visual tags, e.g., visual tags that should be close to its current location, the UAV can widen the search for visual tags or return to its docking station.

In some implementations the UAV can also use an artificial intelligence layer to search for visual tags. The artificial intelligence layer can be trained to process image data collected by the onboard imaging system and identify objects in the image data that are likely to include or be near a visual tag.

For example, as described above, in some implementations visual tags can be positioned on the top of datacenter racks. In this example, the UAV can include an artificial intelligence layer that is trained to identify datacenter racks in an image or video of the datacenter. Then, during the search for a visual tag the UAV can activate the onboard imaging system to collect image data of its current surroundings and process the image data using the trained artificial intelligence layer. If the artificial intelligence layer detects an object, e.g., a datacenter rack, in the image data the UAV can fly towards the detected object to search for a visual tag. For example, if the UAV is currently located in an aisle between two rows of racks, the artificial intelligence layer can determine that there is a datacenter rack directly in front of the UAV. In response, the UAV can increase its altitude so that the top of the datacenter rack and neighboring datacenter racks are visible, and search for a visual tag.

As another example, as described above, in some implementations visual tags can be positioned on the datacenter ceiling. In this example, the UAV can include an artificial intelligence layer that is trained to identify the ceiling or datacenter racks in an image or video of the datacenter. Then, during the search for a visual tag the UAV can activate the onboard imaging system to collect image data of its current surroundings and process the image data using the trained artificial intelligence layer. If the artificial intelligence layer detects an object, e.g., ceiling or datacenter rack, in the image data the UAV can fly towards the detected object or change its field of view to search for a visual tag. For example, if the UAV is currently located in an aisle between two rows of racks, the artificial intelligence layer can determine that there is a datacenter rack directly in front of the UAV. In response, the UAV can determine to adjust its field of view to look upwards at the ceiling and search for a visual tag.

When the UAV identifies a visual tag, the UAV can extract metadata from the visual tag to determine a location and/or other properties of the visual tag (step <NUM>). The UAV then uses the determined location and/or other properties of the visual tag to determine whether to fly to the visual tag or not (step <NUM>). For example, the UAV can determine to fly to a visual tag if the UAV has not recently collected data from the location of the visual tag. Similarly, the UAV can determine not to fly to a visual tag if the UAV has recently, e.g., during the current data collection mission, collected data from the location of the visual tag. As another example, the UAV can determine to fly to a visual tag if the name of the visual tag indicates that the location is relevant to the current data collection mission, e.g., visual tags with names that indicate that the visual tags are located near datacenter rows can be relevant to a datacenter discovery mission whereas visual tags with names that indicate that the visual tags are located near a water treatment facility can be relevant to a mission for checking the operation of the water treatment facility. Similarly, the UAV can determine not to fly to a visual tag if the name of the visual tag indicates that the location is not relevant to the current mission.

In response to determining not to fly to the visual tag at step <NUM>, the example process <NUM> returns to step <NUM> and the UAV uses the onboard imaging system to search for another visual tag.

In response to determining to fly to the visual tag at step <NUM>, the UAV flies towards the visual tag and collects data (step <NUM>). In some implementations the UAV can collect data from a position that is close to, e.g., within a predetermined set of offsets from, the visual tag. For example, if the visual tag is positioned on the ceiling of the datacenter at the end of an aisle that is between two datacenter rows, the UAV can fly to a predetermined distance/offset below the visual tag on the ceiling and collect images (or other sensor data) of the datacenter from this position. This could produce images similar to image <NUM> of <FIG>. In other implementations the UAV can collect data whilst flying from one visual tag to another. For example, if visual the visual tag is positioned on a wall at the end of an aisle that is between two datacenter rows, the UAV can fly along the aisle and collect images (or other sensor data) of the racks in the datacenter rows as it flies along the aisle.

The collected data can include image data and other environmental data. In some implementations image data can be collected using another onboard imaging system that is different to the onboard imaging system used to identify the visual tags. Environmental data can be collected using respective sensors. For example, the UAV can include an anemometer for measuring wind speed, e.g., an amount of wind blowing on racks in the datacenter in order to determine that cooling fans are functioning properly and are not broken or blocked. As another example, the UAV can include a hygrometer for measuring humidity, e.g., in order to determine that a water cooling system is functioning properly and is not leaking. As another example, the UAV can include a thermal sensor to measure temperatures in the datacenter. As another example, the UAV can include a Wi-Fi spectrum analyzer that measures Wi-Fi signals in the datacenter. As another example, the UAV can include a decibel meter or other sensor for measuring noise or sound levels in the datacenter. As another example, the UAV can include an anemometer that measures air flow or wind speed in the datacenter.

In some implementations the UAV can be configured to collect and drop off sensor packages to collect environmental data during the navigation process, e.g., to reduce its weight during the data collection mission. For example, if a visual tag location or name indicates that it is close to a water treatment facility, the UAV can determine to collect a hygrometer to measure air humidity at or near the visual tag. After measuring the air humidity, the UAV can return the hygrometer sensor package. As another example, if a visual tag location or name indicates that it is close to or on a rack, the UAV can determine to collect a thermal sensor to measure a temperature at or near the visual tag. After measuring temperature, the UAV can return the thermal sensor package.

In some implementations, e.g., when wireless connectivity is temporarily unavailable or disabled, the collected data can be stored locally before being uploaded to a server. In other implementations, e.g., when wireless connectivity is available, the collected data can be transmitted to the server in real time. The server can analyze the collected data to monitor the datacenter and predict or prevent failures of components of the datacenter.

Once the UAV has collected the data at step <NUM>, example process <NUM> returns to step <NUM> and the UAV uses the onboard imaging system to search for another visual tag.

In some implementations the UAV can perform the autonomous navigation process in combination with a connectivity-based navigation process to image and sense the datacenter. During a connectivity-based navigation process the UAV can be configured to follow a path in the datacenter and collect data continuously or at certain points along the path. For example, a path through the datacenter can be specified in advance and the UAV can follow the path to complete a data collection mission. The path can be defined in terms of a sequence of waypoints that are selected from a map of the datacenter, where the UAV altitude, speed, and actions (e.g., type of data collection) at or in-between waypoints can be set in advance. The UAV can be configured to follow the path using global positioning system (GPS) and optionally an inertial measurement unit (IMU).

Feedback from the GPS and IMU can enable the UAV to move along the path with minimal error. However, there are several disadvantages of relying on GPS. GPS signals are remote signals and susceptible to disruptions or interference, especially in indoor environments or areas surrounded by high buildings. When a GPS signal is weak or not available the UAV can drift from the path or be unable to complete the mission. In addition, information in a GPS signal can be modified by attackers, which could cause the UAV to follow an incorrect path.

As another example, in some implementations the path can be specified in real time, e.g., via a remote controller operated by a human pilot or a virtual pilot that is programmed to send commands using remote control in real-time as a human pilot would. In these implementations the UAV requires a constant wireless connection to the remote controller and a GPS connectivity.

However, the UAV could lose connection to the remote controller for several reasons - the remote controller could shut down due to a hardware or software issue, the distance between the UAV and remote controller could exceed a connectivity threshold, magnetic interference from the surrounding area could interfere with the connection, or new updates or firmware can disrupt the connection.

In conventional systems, if an UAV loses connectivity during a connectivity-based navigation process, the UAV can perform recovery operations such as: hover in place until connectivity is recovered, land in place until connectivity is recovered, return to its docking station, or backtrack its recent flightpath until connectivity is recovered or until it reaches its previous docking station. Typically, once the drone returns to the docking station or otherwise aborts the mission, it is not possible to restart the mission at the point at which the mission was previously disconnected - the mission must be repeated from the start. These recovery operations can therefore be very inefficient and resource intensive, particularly when considering the typical size of datacenters.

The autonomous datacenter aerial imaging and sensing system described in this specification can implement an improved recovery operation in the case of a loss of connectivity and/or becoming lost, as described below with reference to <FIG> and <FIG>.

<FIG> show examples of environments in which an autonomous datacenter aerial imaging and sensing system executes a connectivity-enabled navigation process and an autonomous navigation process. As used in this specification, a "connectivity-enabled navigation process" is a navigation process by which a UAV navigates using network data received over network link, such as a GPS link, a Wi-Fi link, or some other network link. As shown in <FIG>, a UAV <NUM> in the environment <NUM> executes a connectivity-based navigation process to collect data from a datacenter. For example, the UAV <NUM> can use an available wireless communication <NUM> to communicate with a server and execute a waypoint flight mode. Under the waypoint flight mode, the UAV <NUM> can follow a preprogrammed flight path <NUM> and collect data from the data center along the way.

However, as described above, the UAV <NUM> can lose connection to the server and/or stop receiving GPS signals for a variety of reasons. For example, in the environment <NUM> shown in <FIG> the wireless communication <NUM> is down. The UAV <NUM> can therefore no longer continue the connectivity-enabled navigation process and cannot follow the remainder of the preprogrammed flight path <NUM>. Instead of hovering, landing, or backtracking the flight path <NUM>, the UAV <NUM> switches to an autonomous flight mode and executes an autonomous navigation process until the wireless communication <NUM> returns (or until the data collection mission is complete). As used in this specification, an "autonomous navigation process" is a navigation process by which a UAV navigates using visual data received by use of an onboard imaging sensor and without the use of data received over network link, such as a GPS link, a Wi-Fi link, or some other network link.

For example, the UAV <NUM> can lower its current speed and activate an onboard imaging system. The UAV <NUM> can hover or creep forward at a safe, low speed and, using the onboard imaging system, search <NUM> for visual tags that have been positioned throughout the datacenter, as described above with reference to <FIG> and <FIG>. In example environment <NUM> the UAV <NUM> uses the onboard imaging system to find and identify visual tag <NUM> which has been positioned on the floor in the datacenter.

As described above with reference to <FIG> and <FIG>, the visual tags can be associated with respective logical locations in the datacenter. The UAV <NUM> can use these logical locations to identify its current location and navigate through the datacenter, flying from one visual tag to another. As shown in environment <NUM> of <FIG>, after losing connectivity the UAV <NUM> has executed an autonomous navigation process and, through sequential identification of visual tags, e.g., visual tags <NUM>, <NUM>, and <NUM>, has followed path <NUM> through the datacenter, collecting data along the way.

As shown in environment <NUM> of <FIG>, when the wireless connection <NUM> returns the UAV <NUM> can continue the connectivity-enabled navigation process and follow the remainder of the preprogrammed flight path, e.g., path <NUM>. In some implementations the UAV <NUM> can also perform a reconciliation of the data collection mission. For example, the UAV <NUM> can determine a location at which the connection to the server was lost and a location at which the connection to the server returned. The UAV <NUM> can then determine whether waypoints or data collection steps were missed during the autonomous navigation process between these locations and adjust the preprogrammed flight path included in the connectivity-based navigation process to include these waypoints or data collection steps. As another example, the UAV <NUM> can determine whether additional waypoints and data collection was performed during the autonomous navigation process and adjust the preprogrammed flight path included in the connectivity-based navigation process to remove these waypoints or data collection steps and to avoid repeating a section of the flight path and collecting redundant data.

<FIG> is a flowchart of an example process <NUM> for collecting data from a datacenter using a connectivity-based navigation process and an autonomous navigation process. For convenience, the process <NUM> will be described as being performed by a UAV that is included in an autonomous datacenter aerial imaging and sensing system. For example, the UAV <NUM> of <FIG>, appropriately programmed, can perform example process <NUM>.

The UAV executes a connectivity-enabled navigation process within a data center to collect data from the data center (step <NUM>). During the connectivity-enabled navigation process the UAV is in data communication with a server or controller. Example connectivity-enabled navigation processes are described above with reference to <FIG>.

During execution of the connectivity-enabled navigation process, the UAV determines that connection to the server is lost so that the connectivity-enabled navigation process cannot be continued (step <NUM>). In response to determining that connection to the server is lost, the UAV executes an autonomous navigation process, e.g., as described above with reference to <FIG> and <FIG>, until the connection to the server is recovered (step <NUM>).

Under the autonomous navigation process, the UAV navigates the data center using visual tags positioned in the data center. As described above with reference to <FIG>, to execute the autonomous navigation process the UAV can activate an onboard imaging system and use the onboard imaging system to search for a visual tag positioned in the datacenter. As described above with reference to <FIG>, the visual tags can include geotagged visual media with associated metadata that can include data representing a physical tag location, a logical tag location, tag altitude, or tag name. The visual tags can be positioned independent of pre-defined data collection mission or path, e.g., on the floor of the datacenter, walls of the datacenter, racks at ends of datacenter rows, a ceiling of the datacenter, or a top surface of racks included in the datacenter. Searching a datacenter for a visual tag is described above with reference to <FIG> and <FIG>.

In response to a successful identifying a visual tag, the UAV can extract metadata from the visual tag to determine properties of the visual tag, e.g., a location or name of the visual tag, and use the properties to determine whether to fly to the visual tag or not.

For example, the UAV can determine whether the location of the visual tag is close to, e.g., within a predetermined distance/offset from, a known location from which to collect data or not and in response to determining that the location of the visual tag corresponds to a location from which to collect data, fly to the visual tag. As another example, the UAV can determine whether data has already been collected from the location of the visual tag during a current data collection mission or not and in response to determining that data has not already been collected from the location of the visual tag, fly to the visual tag. As another example the UAV can determine whether a name of the visual tag indicates that the location of the visual tag is relevant to the current data collection mission or not and in response to determining that the name of the visual tag indicates that the location of the visual tag is relevant to the current data collection mission, fly to the visual tag.

When the UAV determines to fly to a visual tag, it can collect data from a position that is within a predetermined set of distance offsets from the visual tag or collect data whilst flying towards the visual tag. As described above with reference to <FIG> and <FIG>, in some implementations the UAV can also use properties of a visual tag to determine whether to collect a sensor package for collecting sensor data from a position that is within a predetermined set of distance offsets from the visual tag. In response to determining to collect a sensor package, the UAV can collect the sensor package and use the sensor package to collect sensor data from a position that is within a predetermined set of distance offsets from the visual tag. After the sensor data has been collected, the UAV can return the sensor package. Example sensors and sensor data are described above with reference to <FIG> and <FIG>.

The UAV can locally store image data or sensor data collected during the autonomous navigation process and upload the stored data to the server when the connection to the server is recovered or when the UAV completes the data collection mission and returns to its docking station.

In some implementations, during execution of the autonomous navigation process, the UAV can determine that the connection to the server has returned so that the connectivity-enabled navigation process can be continued (step <NUM>). In response, the UAV can continue execution of the connectivity-enabled navigation process (step <NUM>). In some implementations the UAV can perform a reconciliation of the connectivity-enabled navigation process before executing the connectivity-enabled navigation process.

For example, in response to determining that the connection to the server has returned, the UAV can determine a location at which the connection to the server was lost and a location at which the connection to the server returned. The UAV can then determine whether waypoints or data collection steps were missed during the autonomous navigation process between the location at which the connection to the server was lost and the location at which the connection to the server returned. In response to determining that waypoints or data collection steps were missed, the UAV can adjust the connectivity-enabled navigation process to include the missed waypoints or data collection steps.

As another example, in response to determining that the connection to the server has returned, the UAV can determine a location at which the connection to the server was lost and a location at which the connection to the server returned. The UAV can then determine whether additional waypoints or data collection steps were performed during the autonomous navigation process. In response to determining that additional waypoints or data collection steps were performed during the autonomous navigation process, the UAV can adjust the connectivity-enabled navigation process to remove the additional waypoints or data collection steps.

<FIG> is a block diagram of an example autonomous datacenter aerial imaging and sensing system <NUM>. The example system <NUM> includes an unmanned aerial vehicle (UAV) <NUM>, a server <NUM>, visual tags <NUM>, and optionally one or more sensors <NUM>. In this example system the sensors <NUM> are separate to the UAV <NUM>, however in some implementations the sensors <NUM> can be included in the UAV <NUM>.

The UAV <NUM> can be, e.g., an industrial drone that is configured to perform a connectivity-enabled navigation process and/or the autonomous navigation process described in this specification. For example, when a connection to the server <NUM> is available the UAV <NUM> can be remotely piloted or programmed to follow a pre-defined flight path. However, when a connection to the server <NUM> is not available the UAV <NUM> can perform an autonomous navigation process as described above with reference to <FIG>.

The UAV <NUM> can include imaging equipment and sensors for collecting data from a datacenter. For example, the UAV <NUM> can include a first imaging system 510a, e.g., electronic imaging device, that is configured to collect image data. In some implementations the UAV <NUM> can also include sensors <NUM> for collecting environmental data from the datacenter. In other implementations sensors <NUM> for collecting environmental data can be external to the UAV <NUM>, as shown in example system <NUM>. In these implementations the UAV <NUM> can be configured to collect, carry, and return sensors for specific data collection missions. The sensors <NUM> can include hygrometers for measuring humidity, thermal sensors for measuring temperatures, anemometers for measuring wind speed, or any other sensor that can collect relevant data from the datacenter.

The UAV <NUM> can also include an image and sensor data store <NUM> that is configured to store data collected by the first imaging system 510a and the sensors <NUM>. For example, the UAV <NUM> can store collected data in the image and sensor data store <NUM> and transmit the stored data to the server <NUM> at regular time intervals, e.g., when a wireless connection to the server <NUM> is available.

The UAV <NUM> can also include a second imaging system 510b, e.g., electronic imaging device, for navigating the datacenter as part of the autonomous navigation process. As shown in example system <NUM>, the second imaging system 510b can be separate to the first imaging system 510a used to collect image data from the data center. The second imaging system 510b can collect image data and provide the collected image data to the processor <NUM>.

The processor <NUM> can be configured to identify visual tags in the image data collected by the second imaging system 510b. If a visual tag is identified, the processor <NUM> can analyze or read the visual tag and extract metadata from the visual tag to determine a location and/or other properties of the visual tag. The processor <NUM> can then determine whether to fly to the visual tag or not, as described above with reference to <FIG>. In response to determining not to fly to the visual tag, the processor <NUM> can analyze other portions of the image data collected by the second imaging system 510b to search for a different visual tag or cause the aerial camera to collect additional image data. In response to determining to fly to the visual tag, the processor <NUM> can instruct the UAV to fly towards the visual tag and collect data.

The UAV <NUM> can further include an artificial intelligence layer <NUM> that is configured to process image data collected by the imaging system 510a and identify objects in the image data. For example, the artificial intelligence layer <NUM> can be trained to detect objects that are likely to include or be near visual tag in order to assist the UAV when searching for a visual tag, as described above with reference to <FIG>.

The UAV <NUM> can further include a reconciliation module <NUM>. The reconciliation module <NUM> is configured to perform reconciliations of data collection missions, e.g., after the UAV <NUM> has performed an autonomous navigation process. For example, the reconciliation module <NUM> can determine whether waypoints or data collection steps were missed during an autonomous navigation process and adjust a preprogrammed flight path included in a connectivity-based navigation process to include these waypoints or data collection steps. Example operations performed by the reconciliation module <NUM> are described above with reference to <FIG>.

The UAV <NUM> and the server <NUM> can be connected via a network, e.g., a local area network (LAN), wide area network (WLAN), the Internet, or a combination thereof, which can be accessed over a wired and/or a wireless communications link. During a connectivity-enabled navigation process, the server <NUM> can remotely pilot the UAV <NUM> or transmit instructions to follow a pre-programmed flight path. The server <NUM> can also receive image or sensor data collected by the UAV <NUM>, e.g., in real time, at regular time intervals, or after a data collection mission is complete. The server <NUM> can analyze received image or sensor data to monitor the datacenter and predict or prevent failures of components of the datacenter.

The visual tags <NUM> can include geotagged visual media, e.g., geotagged photographs, images or QR codes, that are positioned in the datacenter. Metadata represented by the visual tags can include, e.g., a physical tag location, logical tag location, tag altitude, and tag name.

Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus.

<FIG> is a block diagram of computing devices <NUM>, <NUM> that may be used to implement the systems and methods described in this document, as either a client or as a server or plurality of servers. Computing device <NUM> is intended to represent various forms of digital computers or processors, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, smartwatches, head-worn devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations described and/or claimed in this document.

The processor <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM>. The processor may also include separate analog and digital processors.

The display <NUM> may be, for example, a TFT LCD display or an OLED display, or other appropriate display technology. External interface <NUM> may provide, for example, for wired communication (e.g., via a docking procedure) or for wireless communication (e.g., via Bluetooth or other such technologies).

Expansion memory <NUM> may also be provided and connected to device <NUM> through expansion interface <NUM>, which may include, for example, a SIMM card interface.

The memory may include for example, flash memory and/or MRAM memory, as discussed below. The information carrier is a computer- or machine-readable medium, such as the memory <NUM>, expansion memory <NUM>, or memory on processor <NUM>.

Communication interface <NUM> may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA4000, or GPRS, among others. In addition, GPS receiver module <NUM> may provide additional wireless data to device <NUM>, which may be used as appropriate by applications running on device <NUM>.

For example, it may be implemented as a cellular telephone.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the scope of the invention. For example, various forms of the flows shown above may be used, with steps re-ordered, added, or removed. Also, although several applications of the payment systems and methods have been described, it should be recognized that numerous other applications are contemplated. Accordingly, other embodiments are within the scope of the following claims.

Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub combination or variation of a sub combination.

Claim 1:
A computer-implemented method, comprising:
executing, within a datacenter and by an unmanned aerial vehicle, UAV, in data communication with a server, a connectivity-enabled navigation process via a remote controller to collect data from the datacenter;
determining, during execution of the connectivity-enabled navigation process via the remote controller and by the UAV, that connection to the server is lost so that the connectivity-enabled navigation process cannot be continued via the remote controller; and in response to determining that connection to the server is lost, executing, by the UAV, an autonomous navigation process until the connection to the server is recovered, wherein executing the autonomous navigation process comprises:
activating an onboard imaging system; and
searching, using the onboard imaging system, for a visual tag positioned in the datacenter, wherein searching for a visual tag positioned in the datacenter comprises querying an onboard processing system for a visual tag that is likely to be close to a current location of the UAV and flying towards the visual tag; and
in response to successfully identifying, by the onboard imaging system, the visual tag, flying, by the UAV towards the visual tag.