Methods and devices for heart rate controlled drones

A method for controlling a drone including performing operations on a processor configured to control location of the drone are described. The operations on the processor include receiving heart rate messages from a remote device carried by a user, where each heart rate message includes heart rate information of the user, and receiving location messages from the remote device carried by the user, where each location message includes location information of the user. The method includes predicting a future location of the user based on the heart rate messages and the location messages, generating a target location to which the drone is to be moved based on the future location of the user, and commanding the drone to move to the target location. Related devices are disclosed.

BACKGROUND

The present inventive concepts generally relate to controlling a drone.

Running is a global trend that is popular as an exercise routine to improve health, stamina, and general well being of individuals. A multi-billion dollar industry related to running has developed that includes shoes, clothes, wearable devices, fitness tracking devices, apps, etc. In parallel, an industry related to drones has developed. Drones may be an integral part of the future Internet of Things (IoT) enabled world that assist individuals with tasks such as monitoring, filming, cleaning, repairs, and security. These two industries, namely running and drones, may be combined such that drones may be used to improve the experiences of runners in a variety of settings.

SUMMARY

Some embodiments of the present disclosure are directed to a method of controlling a drone by performing various operations on a processor configured to control location of the drone. These operations may include receiving a plurality of heart rate messages from a remote device carried by a user, each heart rate message of the plurality of heart rate messages including heart rate information of the user, and/or receiving a plurality of location messages from the remote device carried by the user, each location message of the plurality of location messages including location information of the user. The operations on the processor may include predicting a future location of the user based on the heart rate messages and the location messages, generating a target location to which the drone is to be moved based on the future location of the user, and/or commanding the drone to move to the target location.

According to various embodiments, predicting the future location of the user based on the heart rate messages and the location messages may include predicting the future location of the user based on differences in the heart rate information in at least two of the heart rate messages and differences in the location information in at least two of the location messages. Predicting the future location of the user may include predicting the future location of the user based on a pace of the user. Generating a target location to which the drone is to be moved may include generating the target location based on the pace of the user.

According to various embodiments, the operations on the processor may include determining the pace of the user based on recorded historical measurements of the user's pace over defined time intervals within a threshold distance of a present location of the user. The operations may include determining the pace of the user based on a mathematical combination of heart rates indicated by at least two of the plurality of heart rate messages that were previously received. Operations may include scaling the pace of the user by a weighting factor defined based on whether the heart rates have increased or decreased over a defined interval.

According to various embodiments, the operations on the processor may include receiving environment information associated with the location information of the user, and/or determining an adjusted pace of the user based on the environment information and the pace of the user. Predicting the future location of the user further may include predicting the future location of the user based on the adjusted pace. The environment information may include weather conditions, terrain information, geographical features, and/or a location of another user. The environment information may indicate presence of persons in a proximity of the user based on detecting radio frequency signals received from devices carried by the persons. The method may include scaling the adjusted pace of the user by a weighting factor in response to determining the presence of persons in the proximity of the user.

According to various embodiments, the operations on the processor may include determining, based on the differences in the heart rate information, that the user is maintaining a steady heart rate, determining, based on the differences in the location information, that the user is stationary, and/or controlling the location of the drone to be stationary based on the determining that the user is maintaining a steady heart rate and the determining that the user is stationary. The operations may include determining, based on the differences in the heart rate information, that the user is accelerating or decelerating, predicting, based on the determining that the user is accelerating or decelerating, the future location of the user, and/or controlling a speed of the drone based on the determining that the user is accelerating or decelerating and the predicting the future location of the user. In some embodiments, determining, based on the differences in the heart rate information, that the user is accelerating or decelerating may include determining that the user is accelerating based on a first heart rate information of one of the plurality of heart rate messages being greater than a second heart rate information of a previous one of the plurality of heart rate messages, and/or determining that the user is decelerating based on the first heart rate information being less than the second heart rate information. The controlling the speed of the drone may be based on the determining that the user is accelerating or decelerating by performing operations including increasing the speed of the drone as the drone travels to the future location of the user that was predicted, in response to the determining that the user is accelerating, and/or decreasing the speed of the drone as the drone travels to the future location of the user that was predicted, in response to the determining that the user is decelerating.

According to various embodiments, the operations may include providing an input to a camera associated with the drone, in response to the predicting the future location of the user based on the differences in the heart rate information and the differences in the location information. The method may include controlling a zoom of the camera, in response to the predicting the future location of the user based on the differences in the heart rate information and the differences in the location information. Controlling the zoom of the camera may include determining, based on differences in the heart rate information in at least two of the heart rate messages, that the user is accelerating or decelerating, decreasing the zoom of the camera responsive to determining that the user is accelerating, and/or increasing the zoom of the camera responsive to determining that the user is decelerating.

According to various embodiments, the operations may include adjusting a field of view of the camera, in response to predicting the future location of the user based on the differences in the heart rate information and the differences in the location information. Adjusting the field of view of the camera further may include determining, based on differences in the heart rate information in at least two of the heart rate messages, that the user is accelerating or decelerating, increasing the field of view of the camera responsive to determining that the user is accelerating, and/or decreasing the field of view of the camera responsive to determining that the user is decelerating. Controlling the location of the drone may include providing information related to the altitude, speed, yaw, roll, pitch, and/or heading of the drone.

Other methods and devices, according to embodiments of the present disclosure will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such methods and devices be included within this description, be within the scope of the present inventive subject matter, and be protected by the accompanying claims.

DETAILED DESCRIPTION

Various embodiments will be described more fully hereinafter with reference to the accompanying drawings. Other embodiments may take many different forms and should not be construed as limited to the embodiments set forth herein. Like numbers refer to like elements throughout. Numerous specific details are set forth in order to provide a thorough understanding of embodiments of the present inventive concepts. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components and circuits have not been described in detail so as not to obscure the present invention. It is intended that all embodiments disclosed herein can be implemented separately or combined in any way and/or combination.

As noted above, running is a popular leisure activity for active individuals who seek to improve their health. Maintaining a good pace and/or maintaining a particular heart rate are goals of many runners. Drones may be able to assist runners improve their experience. Various embodiments described herein may arise from a recognition for a need to improve the experience of runners using drones to provide guidance regarding pacing, direction, filming, etc. The location and/or heart rate of the runner may be used to control the location of the drone such that the drone moves to a position suitable to the runner.

Referring now toFIG. 1, a runner (i.e. user)102is illustrated in view of a drone101that assists and/or guides the runner. The runner102may be carrying a remote device103that is remote from the drone101. The remote device103may measure the heart rate of the user102and/or may determine the location of the user by use of GPS information. Although illustrated as a watch-like device worn on the wrist of the user102, the remote device103may be part of or in communication with a mobile device104carried in the pocket of the user, on a belt clip, or in an armband. In some embodiments, the mobile device104may perform the functions described herein. The remote device103may also include a chest strap worn by the user102for accurate pulse measurements. In some embodiments, the remote device103may include a separate pulse monitoring device that is separate from a location tracking device associated with the user102. In some embodiments, the pulse monitor device may be co-located with the drone101and may be configured to remotely measure the pulse using an infrared (IR) camera and/or by analyzing color changes or motion changes related to the pulse of the user102.

Referring toFIG. 2, a flowchart of operations that may be performed to control the drone101ofFIG. 1is illustrated. These operations may be executed by a processor in the remote device103associated with the user, another device associated with the user, and/or by a processor in the drone101. In some embodiments, these operations may be performed by an application in the remote device103and/or in an associated mobile device104carried by the user. Resulting information regarding a target location may be communicated to the drone101by a wireless link from the remote device103and/or mobile device104ofFIG. 1. As illustrated inFIG. 2, at block201, a plurality of heart rate messages may be received by a remote device103and/or a mobile device104carried by the user102. A plurality of location messages may be received from the remote device103and/or mobile device104ofFIG. 1at block202. The location information and the heart rate information may be in a single message or in different messages, in various embodiments. Although discussed herein in the context of heart rate messages and/or location messages, heart rate information and/or location information may be received in any form of communication from the remote device103and/or a mobile device104carried by the user102.

Still referring toFIG. 2, at block203, a future location of the user may be predicted based on the heart rate messages and the location messages. A target location to which the drone is to be moved may be generated at block204, based on the future location of the user. At block205, the drone may be commanded to move to the target location that was generated. Commanding the drone to move to the target location may include generating information related to the altitude, speed, yaw, roll, pitch, and/or heading to which the drone is to reposition.

Although the discussion herein is based on, for example, control of a drone based on heart rate information, other measurements of physical performance may be used in addition to, or in lieu of, the heart rate measurements. For example, heart rate variability, blood pressure, blood oxygen level, breathing rate, pulse, perspiration, hemoglobin level, electromyography (EMG), electrodermal activity (EDA), skin conductance, galvanic skin response (GSR), electrodermal response (EDR), psychogalvanic reflex (PGR), skin conductance response (SCR), skin conductance level (SCL), and/or other physiological measurements may be used to control the drone. Sensors for brain activity related to exercise performance and/or motivation such as electroencephalogram (EEG), functional magnetic resonance imaging or functional MRI (fMRI), Electrocorticography (ECoG), or intracranial electroencephalography (iEEG), diffusion MRI (dMRI), and/or near-infrared spectroscopy (NIRS) may be used to control the drone.

Referring now toFIG. 3, the future location of the user may be predicted, based on differences in the heart rate information in at least two of the heart rate messages and/or differences in the location information in at least two of the location messages. For example, the heart rate in a recent heart rate message N may be compared to the heart rate in a previous message N−1. Similar comparisons may be made with location information in location messages N and N−1. The present inventive concepts may be extended to the heart rate at N−2, N−3, etc. Based on these two or more samples of the heart rate of the user and/or the location of the user, a pace of the user may be determined. Although consecutive messages N and N−1 are described as an example, any two messages with heart rate and/or location information may be used for determine the differences. In some embodiments, a targeted pace may be set as desired by the user and used for predicting the future location of the user.

Now referring toFIG. 4, based on two or more samples of the heart rate and/or location of the user, the future location of the user may be predicted based on the pace of the user, at block401. Referring toFIG. 5, the target location may be generated based on the pace of the user, at block501. The target location may be a location to which the drone is to move such that the user may follow the drone to maintain a present pace or change to a target pace. In some embodiments, the drone may move to a location in order to increase or decrease the heart rate of the user, based on an optimum training pace of the user or based on a goal of the user. The target location may be determined to provide a suitable view from a camera associated with the drone for purposes of filming.

Referring now toFIG. 6, based on recorded historical measurements of the user's pace over defined time intervals, the pace of the user may be determined, at block601. The pace may be determined based on two or more time intervals within which a user travels at least a threshold distance from the present location of the user. In some embodiments, a historical pace in a particular geographic region may be used to set a target pace that the user strives to achieve. Referring now toFIG. 7, the pace of the user may be determined based on a mathematical combination of heart rates indicated by at least two of the plurality of heart rate messages that were previously received, at block701. The mathematical combination of heart rates may include averaging two or more heart rate measurements, removing outlier heart rate values, and/or determining a mode of a plurality of heart rate values.

Referring now toFIG. 8, the pace of the user may be scaled by a weighting factor that is defined based on whether the heart rates have increased or decreased over a defined time interval, at block801. In other words, the pace of the user may be determined to have accelerated by determining that the heart rates have increased over a defined time interval. Similarly, the peace of the user may be determined to have decelerated in determining that the heart rates have decreased over a defined time interval. The pace of the user may be scaled upward by a weighting factor after a threshold number of increased heart rate values are received. The pace of the user may scaled downward by a weighting factor after a threshold number of decreased heart rate values are received.

Referring now toFIG. 9, environment information associated with the location information of the user may be received, at block901. At block902, an adjusted pace of the user may be determined based on the environment information and/or the pace of the user. The future location may be predicted based on the adjusted pace of the user. The environment information may include weather conditions, terrain information, geographical features, and/or the location of one or more other users. For example, if weather information for the location of the user indicates that it is raining, the adjusted pace of the user may be reduced to take into account difficulty in running on a wet surface, poorer visibility, and/or an added weight of wet clothes. In some embodiments, geographical features such as rocky or sandy running terrain may be determined and use as input to a computation to affect the adjusted pace of the user since these surfaces may offer poorer traction/footing for the user. Other geographical features such as a parking lot may provide a lot of roadblocks for the runner when the parking lot has many cars, slowing the pace. On the other hand, the day of the week or the time of day may also be taken into account. For example, a parking lot may be almost full during business hours on weekdays but may be almost completely empty on a Sunday. The pace of the runner may be adjusted based on information related to the time of day and/or day of the week in conjunction with the heart rate information.

According to various embodiments, once the heart rate is obtained, it may be determined if the heart rate should be increased, decreased, and/or maintained at the current level. The environment of the user may be analyzed as previously discussed. A user's heading may be determined based on the current location/heading of the user, a predetermined route for the user, and/or based on a probability that the user will follow the predetermined route. The probability that the user will follow the predetermined route may be affected by factors related to the environment information. The drone may be commanded to increase speed in order to increase the user's heart rate, maintain a current speed to keep a current pace and/or heart rate, and/or decrease speed in order to decrease the heart rate. The drone may be commanded to decrease speed based on environment information such as, for example, if a hill is approaching but the heart rate of the user needs to be maintained.

Referring now toFIG. 10, the environment information may indicate the presence of persons in a proximity of the user based on detecting radio frequency signals received by the mobile device104from devices carried by the other persons. In some embodiments, an indication of presence of persons in the proximity of the user may be received from a network. A density of other users may indicate runners in a race or a high pedestrian traffic area. The adjusted pace of the user may be scaled by a weighting factor in response to determining at least a threshold number of mobile devices from which RF signals are detected in the proximity of the user, at block1001.

In some embodiments, a traffic light or crosswalk may be in the path of the runner. Referring now toFIG. 11, based on the differences in the heart rate information, it may be determined that the user is maintaining a steady heart rate, at block1101. Based on different samples of location information, it may be determined that the user is stationary, at block1102. For example, a runner may be at a traffic light but is jogging in place to maintain a steady heart rate. At block1103, the location of the drone may be controlled to be stationary based on determining that the user is maintaining a steady heart rate and determining that the user is stationary. The drone may be instructed to move once it is determined that the user is no longer stationary.

Heart rate and/or location information may be used to determine that a runner is accelerating or decelerating. Acceleration and deceleration may be due to hills and/or inclines/declines in the running course. Acceleration and deceleration of the heart rate may occur due to changes in the energy level of the runner at different points during the run. Referring now toFIG. 12, based on the differences in the heart rate information, it may be determined that the user is accelerating or decelerating, at block1201. The future location of the user may be estimated based on the recognition that the user is accelerating or decelerating, at block1202. The speed of the drone may be controlled based on the determination that the user is accelerating or decelerating and the predicted future location of the user, at block1203.

Referring now toFIG. 13, determining that the user is accelerating or decelerating based on the differences in the heart rate information may include determining that the user is accelerating based on a heart rate information of a given heart rate message being greater than the heart rate information of a previous one or more of the heart rate messages, at block1301. The user may be decelerating if it is determined that the heart rate information of a given heart rate message is less than the heart rate information of a previous one or more of the heart rate messages, at block1302.

Referring now toFIG. 14, the speed of the drone may be controlled based on the determining that the user is accelerating or decelerating. The speed of the drone may be increased as the drone travels to the future location of the user that was predicted, if it is determined that the user is accelerating, at block1401. The speed of the drone may be decreased as the drone travels to the future location of the user that was predicted, if it is determined that the user is decelerating, at block1402.

The drone may be controlled to move smoothly, controlled to maintain a defined distance and/or azimuth angle relative to the user by controlling the drone based on the heart rate. For example, the user can be expected to be able to have greater likelihood of high acceleration in speed when the user's heart rate is below a defined threshold. In contrast, the user can be expected to have a greater likelihood of deceleration in speed when the user's heart rate is above a defined threshold. In some embodiments, upper and/or lower threshold heart rate values may be defined in order to command the drone to increase or decrease distance from the user. If a heart rate measurement is above an upper threshold heart rate, the drone may be commanded to decrease distance between the user and the drone since it is anticipated that the user has increased speed but will not be able to sustain the higher speed and will soon decelerate. If a heart rate measurement is below a lower threshold heart rate, the drone may be commanded to increase distance between the user and the drone since it is anticipated that the user has decreased speed and will have reserve energy to be able to accelerate in the near future.

In some embodiments, a camera associated with the drone may be controlled based of the heart rate of the user. Referring now toFIG. 15, an input to a camera associated with the drone may be provided, in response to predicting the future location of the user based on the differences in the heart rate information and/or the differences in the location information, at block1501. Referring now toFIG. 16, a zoom of the camera may be controlled, in response to predicting the future location of the user based on the differences in the heart rate information and/or the differences in the location information, at block1601. Referring now toFIG. 17, the zoom of the camera may be controlled to maximize the view of the runner. Based on differences in the heart rate information in at least two of the heart rate messages, it may be determined that the user is accelerating or decelerating, at block1701. The zoom of the camera may be decreased, if the user is accelerating, at block1702. The zoom of the camera may be increased, if the user is decelerating, at block1703. In other words, if the runner is slowing down, a closer view of the runner may be possible by adjusting the zoom of the camera.

In some embodiments, the field of view of a camera associated with the drone may be adjusted based of the heart rate of the user. The field of view, also referred to as “angle of view (AOV)”, describes the angular extent of a given scene that is imaged by the camera. Referring now toFIG. 18, the field of view of the camera may be adjusted, in response to the predicting the future location of the user based on the differences in the heart rate information and/or the differences in the location information, at block1801. Referring now toFIG. 19, adjusting the field of view of the camera may include determining, based on differences in the heart rate information in at least two of the heart rate messages, that the user is accelerating or decelerating, at block1901. The field of view of the camera may be increased if the user is accelerating, at block1902. The field of view of the camera may be decreased if the user is decelerating, at block1903.

As discussed herein, the heart rate allows prediction of how quickly the user may speed up. The drone can be repositioned to allow the user to stay within the field of view of the camera if the predicted speed-up occurs. Due to the inaccuracy of determining GPS location by mobile devices over small distances (for example, 15 feet), controlling a drone based on GPS location alone can lead to erratic control movements and result in the drone not being properly positioned to provide a target distance and/or azimuth angle relative to the direction of movement of the user. When the drone is photographing or capturing video of the user, there is a risk that the user will leave the field of view of the camera. Moreover, erratic control of the drone can result in undesirable erratic movement of the camera. Embodiments of the present disclosure use the heart rate of the person to predict that the person is likely to accelerate or decelerate in order to improve control of the drone.

FIG. 20is a block diagram of a device2000, for use in conjunction with drone101ofFIG. 1, that is configured according to one or more embodiments disclosed herein. The device2000can include a transceiver2030, a network interface2020, a processor circuit2002, and a memory circuit2010containing computer readable program code2012.

The transceiver2030is configured to communicate with the drone101ofFIG. 1using one or more of the radio access technologies. The processor circuit2002may include one or more data processing circuits, such as a general purpose and/or special purpose processor, e.g., microprocessor and/or digital signal processor, that may be collocated or distributed across one or more networks. The processor circuit2002(also referred to as a processor) is configured to execute the computer readable program code2012in the memory2010to perform at least some of the operations and methods of described herein as being performed by the device2000. For example, processor2002may be configured to perform operations discussed above with respect toFIGS. 2-19. The network interface2020communicates with other devices2000, a drone101, a remote device103, and/or a mobile device104carried by the user102ofFIG. 1.

FIG. 21illustrates the computer readable program code2012in more detail. In particular, the computer readable program code2012includes a receiving heart rate module2112, a receiving location module2116, a predicting future location module2120, a generating target location module2124, and/or a commanding drone module2128for commanding the drone101ofFIG. 1to move to the target location.

FURTHER DEFINITIONS AND EMBODIMENTS

Computer program code for carrying out operations for aspects of the present inventive concepts may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages, such as the “C” programming language, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP, dynamic programming languages such as Python, Ruby and Groovy, or other programming languages. The program code may execute entirely on the device, partly on the device, as a stand-alone software package, partly on the device and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider) or in a cloud computing environment or offered as a service such as a Software as a Service (SaaS).

It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting to other embodiments. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense expressly so defined herein.