Patent Description:
Vehicle manufacturers are beginning to offer remote-control driving systems that allow the user to control their vehicles. For example, some manufacturers allow the user to control a vehicle to move from a parked location to a user's desired location. These autonomous driving features allow a vehicle to leave a parking space and navigate around obstacles to its owner. In essence, an owner's vehicle may be "summoned" to you from a threshold distance away. These systems, however, may require line of sight and/or may result in near crashes due to their inability to adjust to changes in the environment. Thus, these systems are limited in that the user can only tell the vehicle to arrive or go to a location, and the vehicle determines the optimal path to the desired location. This results in the user relinquishing control of the path that the vehicle travels to the desired location.

Accordingly, there is a need for a system, apparatus and/or method to increase the amount of user remote control of the vehicle to increase the capability of the remote-control function.

<CIT> refers to automated extraction of semantic information to enhance incremental mapping modifications for robotic vehicles.

<CIT> discloses a remote control platform for a vehicle, comprising: a sensor configured to detect one or more objects within a surrounding environment of the vehicle; and an electronic control unit coupled to the sensor and configured to determine an orientation of the vehicle and a route that includes a direction of travel, and navigate the vehicle along the route based on the one or more objects, the direction of travel and the orientation of the vehicle.

In general, one aspect of the subject matter described in this disclosure may be embodied in a remote control platform for a vehicle. The remote control platform includes a sensor. The sensor is configured to detect one or more objects within a surrounding environment of the vehicle. The remote control platform includes an electronic control unit. The electronic control unit is coupled to the sensor. The electronic control unit is configured to determine an orientation of the vehicle and a route that includes a direction of travel. The electronic control unit is configured to navigate the vehicle along the route based on the one or more objects, the direction of travel and the orientation of the vehicle.

These and other embodiments may optionally include one or more of the following features. The electronic control unit may be configured to determine a current location of the vehicle. The electronic control unit may be configured to categorize the one or more objects within the surrounding environment into a corresponding category of multiple categories. The electronic control unit may be configured to determine a location of each of the one or more objects relative to the vehicle. The electronic control unit may be configured to provide the current location of the vehicle, the corresponding category and the location of each of the one or more objects to a user device to render a graphical representation on the user device.

The electronic control unit may be configured to update the location of each of the one or more objects relative to the vehicle. The electronic control unit may be configured to provide the updated location of each of the one or more objects to track movements of the one or more objects. The graphical representation that is rendered may include one or more icons that represent the current location of the vehicle, the corresponding category and the location of each of the one or more objects. The electronic control unit may be configured to obtain, from a user device, driving instructions. The driving instructions may include the orientation of the vehicle and the route of the vehicle. The electronic control unit may be configured to navigate the vehicle around the one or more objects with the vehicle traveling in the obtained orientation.

The electronic control unit may be configured to obtain, from a user device, driving instructions that include a speed of travel of the vehicle. The electronic control unit may be configured to navigate the vehicle further based on the speed of travel of the vehicle. The electronic control unit may be configured to determine that the one or more objects of the surrounding environment intersect with the route of the vehicle. The electronic control unit may be configured to cancel the navigation of the vehicle along the route of the vehicle. The electronic control unit may be configured to provide a notification that indicates the cancellation of the navigation of the vehicle along the route. The electronic control unit may be configured to provide sensor data of the surrounding environment of the vehicle. The electronic control unit may be configured to provide a suggested alternative route to navigate the vehicle around the one or more objects that intersect with the route of the vehicle.

In another aspect, the subject matter may be embodied in a user device. The user device includes a memory. The memory is configured to store computer-readable instructions that include a software application. The user device includes a processor coupled to the memory. The processor is configured to execute the software application and perform operations to navigate a vehicle. The operations include rendering, on a display, a first location of the vehicle and a first location of each of the one or more objects in proximity to the current location of the vehicle. The operations include obtaining driving instructions including an orientation of the vehicle and a route to navigate the vehicle. The operations include providing the driving instructions to a remote control platform to navigate the vehicle along the route around the one or more objects.

In another aspect, the subject matter may be embodied in a method for navigating an autonomous vehicle. The method includes determining, by a processor, a current location of the vehicle. The method includes detecting, by the processor, one or more objects within a surrounding environment of the current location of the vehicle. The method includes navigating or controlling, by the processor, the vehicle based on a route and an orientation of the vehicle.

Other systems, methods, features, and advantages of the present invention will be apparent to one skilled in the art upon examination of the following figures and detailed description. Component parts shown in the drawings are not necessarily to scale and may be exaggerated to better illustrate the important features of the present invention.

Disclosed herein are systems, apparatuses, and methods to remotely control a vehicle via smartphone and/or gesture input. The remote control system includes a remote control platform and a user device. The remote control platform maps out the locations of the vehicle and one or more objects surrounding the vehicle so that the user can create or generate a route that navigates the vehicle around the one or more objects to the destination. The remote control platform provides the locations of the vehicle and the one or more objects so that the user device may generate a rendering of the environment surrounding the vehicle to allow the user to control the vehicle within the environment in greater detail. The remote control platform provides the user on the user device with greater situational awareness of the environment around the vehicle, which allows the user to provide more detailed driving instructions and more precise control of the vehicle via the user device. For example, user input may specify driving instructions that remotely operate the vehicle to drive <NUM> feet forward, turn left and then drive another <NUM> feet to reach the destination location, rather than simply specifying to arrive at a particular destination.

The more detailed driving instructions allow the user to dictate the route that the vehicle navigates around the one or more objects in the surrounding environment. This allows the user to control the exact path that the vehicle travels to the destination from the starting location. Moreover, the user can control other functions of the vehicle, such as the speed, orientation and direction of travel of the vehicle.

Other benefits and advantages include the capability to monitor the objects in the surrounding environment in real-time. For example, some of the objects may be other vehicles, and the remote control platform may identify when these objects move. If these objects are in the path of or move into the path of the vehicle, the remote control platform may disable or cancel navigation of the vehicle to prevent an accident. The remote control platform may also notify the user, suggest an alternative route or allow for manual control.

Additionally, the remote control platform may identify or recognize visual gestures of a user to remotely control the vehicle. The remote control platform may capture gestures of a user and generate driving instructions based on the gestures to navigate the vehicle along a route. This allows a user to control the vehicle via gesture input.

<FIG> is a block diagram of a remote control system <NUM>. The remote control system <NUM> may be retro-fitted, coupled to, integrated with, include or be included within a vehicle <NUM> or may be entirely separate from the vehicle <NUM>. The remote control system <NUM> may include or be coupled to a remote control platform <NUM>, a user device <NUM> and/or an external database <NUM>. The user device <NUM> may be a personal device, a mobile device, such as a smartphone, a tablet or other electronic device that may display notifications, run applications or otherwise interact with the vehicle <NUM> via a wireless or a wired connection. The remote control system <NUM> uses the user device <NUM> to render a graphical representation of the vehicle <NUM> and the objects surrounding the vehicle <NUM> and to obtain and map out driving instructions to control the vehicle <NUM>. <FIG> is a block diagram of the user device <NUM>.

The remote control system <NUM> may have or use a network <NUM> to communicate among different components, such as between the vehicle <NUM>, the user device <NUM>, the remote control platform <NUM> and/or the external database <NUM>. The network <NUM> may be a Dedicated Short-Range Communication (DSRC) network, a local area network (LAN), a wide area network (WAN), a cellular network, the Internet, or combination thereof, that connects, couples and/or otherwise communicates among the different components of the remote control system <NUM>.

The remote control system <NUM> may include or be coupled to the external database <NUM>. A database is any collection of pieces of information that is organized for search and retrieval, such as by a computer, and the database may be organized in tables, schemas, queries, reports, or any other data structures. A database may use any number of database management systems. The external database <NUM> may include a third-party server or website that stores or provides information. The information may include real-time information, periodically updated information, or user-inputted information. A server may be a computer in a network that is used to provide services, such as accessing files or sharing peripherals, to other computers in the network.

The external database <NUM> may be a map database that includes the locations of various structures or objects, such as trees, buildings or other objects, which interfere with the route of the vehicle <NUM>. The external database <NUM> may be a traffic database. The traffic database may include the locations other vehicles and/or track the locations of the other vehicles, such as movements of the other vehicles.

The remote control system <NUM> includes a remote control platform <NUM>. The remote control platform <NUM> detects or determines the location of the vehicle <NUM> and the one or more objects in the surrounding environment in proximity to the vehicle <NUM>, such as within a threshold distance. The remote control platform <NUM> may detect the movements of the vehicle <NUM> and the movements of the one or more objects and provide the locations, movements and types of objects to the user device <NUM> so that the user device <NUM> may create or generate a rendering of the environment surrounding the vehicle <NUM>. Moreover, the remote control platform <NUM> may receive driving instructions from the user device <NUM> and control or operate the vehicle <NUM> so that the vehicle <NUM> avoids obstacles or hazards to arrive at the designated destination. This allows the user to have more control over the navigation, control and/or operation of the vehicle <NUM> via the user device <NUM> or through gesture input.

The remote control platform <NUM> may include an electronic control unit <NUM>, a memory <NUM>, a network access device <NUM> and/or one or more sensors <NUM>. The remote control platform <NUM> may include a navigation unit <NUM> and/or a user interface <NUM>. The remote control platform <NUM> may include one or more other components of the vehicle <NUM>, such as the brakes <NUM>, the accelerator <NUM>, the steering <NUM>, the motor and/or generator <NUM>, the engine <NUM>, the battery <NUM>, and/or the battery management and control unit (BMCU) <NUM>.

A vehicle <NUM> is a conveyance capable of transporting a person, an object, or a permanently or temporarily affixed apparatus. The vehicle <NUM> may be a self-propelled wheeled conveyance, such as a car, a sports utility vehicle, a truck, a bus, a van or other motor, battery or fuel cell driven vehicle. For example, the vehicle <NUM> may be an electric vehicle, a hybrid vehicle, a hydrogen fuel cell vehicle, a plug-in hybrid vehicle or any other type of vehicle that has a fuel cell stack, a motor and/or a generator. Other examples of vehicles include bicycles, trains, planes, or boats, and any other form of conveyance that is capable of transportation. The vehicle <NUM> may be semi-autonomous or autonomous. That is, the vehicle <NUM> may be self-maneuvering and navigate without human input. An autonomous vehicle may have and use one or more sensors and/or a navigation unit to drive autonomously.

The remote control system <NUM> includes or couples to one or more processors, such as the electronic control unit (ECU) <NUM> of the remote control platform <NUM> or the processor <NUM> of the user device <NUM>. The one or more processors, such as the ECU <NUM> or the processor <NUM>, may be implemented as a single processor or as multiple processors. For example, the ECU <NUM> or the processor <NUM> may be a microprocessor, data processor, microcontroller or other controller, and may be electrically coupled to some or all the other components within the vehicle <NUM> and/or the remote control system <NUM>. The one or more processors may render a graphical representation of the locations of the vehicle <NUM> and/or the objects in the surrounding environment. The one or more processors may also obtain and/or navigate driving instructions inputted by the user. The ECU <NUM> may be coupled to the memory <NUM>, and the processor <NUM> may be coupled to the memory <NUM>.

The remote control system <NUM> has a memory <NUM> within the remote control platform <NUM> and/or a memory <NUM> within the user device <NUM>. The memory <NUM> may be coupled to the ECU <NUM> and store instructions that the ECU <NUM> executes, the memory <NUM> may be coupled to the processor <NUM> and store instructions that the processor <NUM> executes. The memory <NUM> and/or the memory <NUM> may include one or more of a Random Access Memory (RAM), Read Only Memory (ROM) or other volatile or non-volatile memory. The memory <NUM>, <NUM> may be a non-transitory memory or a data storage device, such as a hard disk drive, a solid-state disk drive, a hybrid disk drive, or other appropriate data storage, and may further store machine-readable instructions, which may be loaded and executed by the ECU <NUM> and/or the processor <NUM>. For example, the memory <NUM> may store a software application that renders a graphical representation of the environment surrounding the vehicle <NUM> and may be used to access driving instructions that control or operate the vehicle <NUM>.

The user device <NUM> may include a user interface <NUM> and/or a network access device <NUM>, and the remote control platform <NUM> may include a user interface <NUM> and/or a network access device <NUM>. The one or more user interfaces <NUM>, <NUM> may receive user input that indicates driving instructions to be used to operate, control or otherwise navigate the vehicle <NUM>. The one or more user interfaces <NUM>, <NUM> may also receive configuration tolerances that indicate when an object presents a hazard and intersect into the path or the route of the vehicle <NUM>. The one or more user interfaces <NUM>, <NUM> may also provide notifications and/or a graphical representation of the surrounding environment of the vehicle <NUM> so that the user can input a route to navigate the vehicle <NUM> from a starting location to a destination location.

The one or more user interfaces <NUM>, <NUM> may include an input/output device that receives user input from a user interface element, a button, a dial, a microphone, a keyboard, or a touch screen. The one or more user interfaces <NUM>, <NUM> may provide an output to an output device, such as a display, a speaker, an audio and/or visual indicator, or a refreshable braille display.

The one or more network access device <NUM>, <NUM> may include a communication port or channel, such as one or more of a Dedicated Short-Range Communication (DSRC) unit, a Wi-Fi unit, a Bluetooth® unit, a radio frequency identification (RFID) tag or reader, or a cellular network unit for accessing a cellular network (such as <NUM>, <NUM> or <NUM>). The one or more network access device <NUM>, <NUM> may transmit data to and receive data from the different components of the different entities of the remote control system <NUM>, such as the user device <NUM>, the external database <NUM> and/or the vehicle <NUM>.

The remote control platform <NUM> includes a navigation unit <NUM>. The navigation unit <NUM> may be integral to the vehicle <NUM> or a separate unit coupled to the vehicle <NUM>. The vehicle <NUM> may include a Global Positioning System (GPS) unit (not shown) for detecting location data including a current location of the vehicle <NUM> and date/time information instead of the navigation unit <NUM>. In that regard, the ECU <NUM> may perform the functions of the navigation unit <NUM> based on data received from the GPS unit. The navigation unit <NUM> or the ECU <NUM> may perform navigation functions. Navigation functions may include, for example, route and route set prediction, providing navigation instructions, and receiving user input such as verification of predicted routes and route sets or destinations.

The remote control platform <NUM> includes one or more sensors <NUM>. The one or more sensors <NUM> may include a camera 118a. The camera 118a may capture image data of the surrounding environment, which may be processed or analyzed to recognize structures or objects, such as trees, buildings or other vehicles within the surrounding environment of the vehicle <NUM> and their relative distances from the vehicle <NUM>. The image data may also be used to determine an orientation of the vehicle <NUM>, such as if the vehicle <NUM> is facing in the forward or reverse direction. The one or more sensors <NUM> may include a proximity sensor 118b. The proximity sensor 118b may be use LIDAR, radar or other range-finding technology to detect or determine objects near, such as within a threshold distance, of the vehicle <NUM>. The proximity sensor 118b may also be used to determine the relative distance to those objects. The one or more sensors <NUM> may include other sensors 118c. The other sensors 118c may include a vehicle speed sensor, which may be used to measure the speed of the vehicle <NUM>, and/or an accelerometer or gyroscope, which may be used to determine the orientation of the vehicle <NUM>. The one or more sensors <NUM> may also be used to orient the vehicle <NUM> and to autonomously drive the vehicle <NUM>. For example, the one or more sensors <NUM> may be used to control the vehicle <NUM> to navigate along the route included within the driving instructions. The one or more sensors <NUM> may be used to control the vehicle <NUM> to navigate around the one or more objects and to maintain the relative distance between the vehicle <NUM> and the one or more objects as indicated in the driving instructions and/or to otherwise follow the path or route indicated in the driving instructions.

The remote control system <NUM> may include or be coupled to one or more vehicle components. The remote control system <NUM> may control the one or more vehicle components to operate or otherwise control the vehicle <NUM> to move the vehicle <NUM> from the starting location to the destination location along the route and around the one or more objects in the surrounding environment. The one or more vehicle components may include a motor and/or generator <NUM>. The motor and/or generator <NUM> may convert electrical energy into mechanical power, such as torque, and may convert mechanical power into electrical energy. The motor and/or generator <NUM> may be coupled to the battery <NUM>. The motor and/or generator <NUM> may convert the energy from the battery <NUM> into mechanical power, and may provide energy back to the battery <NUM>, for example, via regenerative braking. The vehicle <NUM> may include one or more additional power generation devices such as the engine <NUM> or a fuel cell stack (not shown). The engine <NUM> combusts fuel to provide power instead of and/or in addition to the power supplied by the motor and/or generator <NUM>.

The battery <NUM> may be coupled to the motor and/or generator <NUM> and may supply electrical energy to and receive electrical energy from the motor and/or generator <NUM>. The battery <NUM> may include one or more rechargeable batteries and may supply the power to the remote control system <NUM> even when the vehicle <NUM> is off.

The BMCU <NUM> may be coupled to the battery <NUM> and may control and manage the charging and discharging of the battery <NUM>. The BMCU <NUM>, for example, may measure, using battery sensors, parameters used to determine the state of charge (SOC) of the battery <NUM>. The BMCU <NUM> may control the battery <NUM>.

The one or more vehicle components may include the brakes <NUM>, the accelerator <NUM> and/or the steering <NUM>. The remote control platform <NUM> may control or operate the brakes <NUM>, the accelerator <NUM> and/or the steering <NUM> to navigate the vehicle <NUM> around the surrounding environment along the route. For example, the remote control platform <NUM> may apply or release the brakes <NUM> or the accelerator <NUM> and/or adjust an angle of the steering <NUM> to decrease the speed, increase the speed and/or change the direction of travel of the vehicle <NUM>.

<FIG> is a flow diagram of a process <NUM> for controlling or operating the vehicle <NUM> using driving instructions. One or more computers or one or more data processing apparatuses, for example, the ECU <NUM> of the remote control platform <NUM> of the remote control system <NUM> of <FIG>, appropriately programmed, may implement the process <NUM>.

The remote control platform <NUM> obtains navigational map information (<NUM>). The navigational map information may include the current location of the vehicle <NUM> and/or the locations of one or more objects in the surrounding environment of the vehicle <NUM>. The remote control platform <NUM> may obtain the navigational map information using the navigation unit <NUM> and/or the external database <NUM>. For example, the remote control platform <NUM> may obtain the current location of the vehicle <NUM> using a global positioning system device and then provide the current the current location of the vehicle <NUM> to the external database <NUM>. The external database may then provide and the remote control platform <NUM> may receive the locations of one or more objects in the surrounding environment of the vehicle <NUM>, such as the location of other vehicles, the location of other structures, such as buildings, and/or the location of other objects, such as trees, which have been mapped.

The remote control platform <NUM> detects or identifies the one or more objects in the surrounding environment of the vehicle <NUM> (<NUM>). The remote control platform <NUM> may outline or frame each of the one or more objects in the surrounding environment of the vehicle <NUM>. The remote control platform <NUM> may compare the outline or frame for each of the one or more objects in the surrounding environment of the vehicle <NUM> to a library of objects and determine whether the object matches one of the library objects. When there is a match, the remote control platform <NUM> may identify the object in the surrounding environment as the library object, such as a tree, a sign or other vehicle, and recognize the object in the surrounding environment, as such.

Additionally, the remote control platform <NUM> may use the one or more sensors <NUM> to determine the location of the one or more objects surrounding the vehicle <NUM>. The location may be a relative location of the one or more objects to the vehicle <NUM>. For example, the remote control platform <NUM> may use the proximity sensor 118b or analyze the image data from the camera 118a to determine the distance and direction of the one or more objects relative to the vehicle <NUM>. The identification of the object along with the location of the object may be associated with each of the one or more objects.

Once the one or more objects are identified or detected, the remote control platform <NUM> may categorize the one or more objects in the surrounding environment of the vehicle <NUM> (<NUM>). When the remote control platform <NUM> matches the one or more objects to a library object, the library object may be associated with a category, such as "building", "tree", "vehicle" or another category, such as "fixed object" or "moving object. " The remote control platform <NUM> may associate the category of the matched library object with that of the object and label or otherwise tag the one or more objects with the label or tag to identify the category of the one or more objects.

The remote control platform <NUM> provides the categorization of the one or more objects, the location of the vehicle <NUM> and/or the locations of the one or more objects within the surrounding environment to the user device <NUM> (<NUM>). The remote control platform <NUM> may send or transmit the categorization of the one or more objects, the location of the vehicle <NUM> and/or the locations of the one or more objects within the surrounding environment to the user device <NUM> through the network <NUM> using the network access device <NUM>. The categorization and/or the locations may be used to generate a rendering of the environment surrounding the vehicle <NUM> so that the user may use the user device <NUM> to navigate or control the vehicle <NUM>.

The remote control platform <NUM> may obtain or detect gesture input (<NUM>). The remote control platform <NUM> may use the one or more sensors <NUM>, such as the camera 118a, to capture the gesture input. For example, the remote control platform <NUM> may capture image data of the surrounding environment and identify an individual, such as the driver or owner, within a threshold distance of the vehicle <NUM>. Then, the remote control platform <NUM> may identify within the image data that the user is moving or that a part of the user, such as the hand of the user is moving. The gesture input may be used to determine driving instructions.

The remote control platform <NUM> may obtain user input (<NUM>). The remote control platform <NUM> may receive the user input from the user device <NUM>. The user input may indicate the driving instructions to navigate or otherwise control the vehicle <NUM>. The user input may be used to determine a speed of travel, a direction of travel, an orientation of the vehicle <NUM>, a starting location of the vehicle <NUM>, a destination location of the vehicle <NUM> and a route from the starting location to the destination location around the one or more objects within the surrounding environment of the vehicle <NUM>. The user input may also indicate the path of the vehicle <NUM> relative to the locations of the one or more objects from the starting location to the destination location.

The remote control platform <NUM> may determine the driving instructions based on the user input provided to the user interface and/or the gesture input (<NUM>). The remote control platform <NUM> may interpret the driving instructions from the user input and/or the gesture input. The remote control platform <NUM> may determine whether to use the user input and/or the gesture input based on a configuration setting that is set by the user beforehand and/or based on an identity of the driver of the vehicle <NUM>, which may be recognized from captured image data by the one or more cameras 118a.

The driving instructions of the vehicle <NUM> may include a speed of travel, a direction of travel, an orientation of the vehicle <NUM>, a starting location of the vehicle <NUM>, a destination location of the vehicle <NUM> and a route from the starting location to the destination location around the one or more objects within the surrounding environment of the vehicle <NUM>. The route may include the path of the vehicle <NUM> relative to the location of the one or more objects.

The remote control platform <NUM> may use a mapping that associates the received user input from the user device <NUM> to driving instructions, such as the speed or direction of travel, the orientation of the vehicle <NUM> and/or the route of the vehicle <NUM>. Similarly, the remote control platform <NUM> may use a mapping that associates the received gesture input to the driving instructions. The driving instructions may convert to or be associated with particular operations or control of the vehicle <NUM>, such as the operation of the brakes <NUM>, the accelerator <NUM> and/or the steering <NUM>.

For example, the remote control platform <NUM> associate a curve in the path indicated in the user input to an angling of the steering <NUM>, which causes the vehicle <NUM> to turn. The angle of the curve may correspond to the amount of angling of the steering <NUM>. In another example, the remote control platform <NUM> may associate a straight line that extends a first length and the speed of the dragging of a finger of user on the user device <NUM> indicated in the user input to the vehicle <NUM> traveling a first distance in a straight line at a particular speed by applying the accelerator <NUM> of the vehicle <NUM>. As the user drags their finger faster on the user device <NUM>, the vehicle <NUM> may accelerate. And as the user drafts their finger farther along in the straight direction, the vehicle <NUM> travels in the straight path for a longer distance. Similarly, when the user stops their finger on the screen of the user device <NUM>, this may indicate to the remote control platform <NUM> to apply the brakes <NUM>. The distance between the path of the user's finger on the display on the user device <NUM> and an icon of an object may directly correspond to the distance between the route of the vehicle <NUM> and the location of the object in the surrounding environment, and so, the path of the user's finger directly mimics or mirrors the actual path of the vehicle <NUM> in the surrounding environment. Moreover, the longer the path that is drawn by the user's finger on the display, the longer the straight path that the vehicle <NUM> travels in the straight path in the surrounding environment.

In another example, the remote control platform <NUM> may recognize handwaving in a particular direction by the user. The handwaving may indicate the direction that the vehicle <NUM> is to travel and the speed of the handwaving may indicate the speed that the vehicle <NUM> is to travel. In another example, the remote control platform <NUM> may recognize the user holding their hand held up above their head, which may indicate to stop the vehicle <NUM> from moving.

In some implementations, the user device <NUM> may receive the user input or gesture input from the user and perform the mapping and/or interpretation of the user input into the driving instructions. The user device <NUM> may perform the mapping and/or interpretation of the user input into the driving instructions and provide the driving instructions to the remote control platform <NUM>. The mapping, interpretation and/or generation of the driving instructions from the user input and/or the gesture input may be learned or otherwise improved upon using machine learning algorithms.

Once the driving instructions are determined, the remote control platform <NUM> determines whether the one or more objects present a hazard to the vehicle <NUM> (<NUM>). The one or more objects may present a hazard to the vehicle <NUM> when the one or more objects intersect or cross into the route of the vehicle <NUM>. The remote control platform <NUM> may compare the route of the vehicle <NUM> included within the driving instructions to the locations of the one or more objects, and if the route of the vehicle <NUM> traverses within a threshold distance of the locations of the one or more objects, the remote control platform <NUM> may determine that the object presents a hazard to the vehicle <NUM>.

Similarly, the remote control platform <NUM> may identify that certain objects of the one or more objects are "moving objects" based on the category of the one or more objects and continually and regularly update the locations of the moving objects. The remote control platform <NUM> may map the path of the one or more moving objects based on the updated locations and determine a trajectory or path of the one or more moving objects. And if the trajectory or the path of the one or more moving objects intersects with the route of the vehicle <NUM>, the remote control platform <NUM> may determine that the object presents a hazard.

If the one or more objects do not present a hazard to the vehicle <NUM>, the remote control platform <NUM> operates or controls the vehicle <NUM> (<NUM>). If the one or more objects do present a hazard to the vehicle <NUM>, the remote control platform <NUM> provides a notification to the user (<NUM>) and/or cancels the navigation or control of the vehicle <NUM> (<NUM>).

When the one or more objects do not present a hazard to the vehicle <NUM>, the remote control platform <NUM> operates or controls the vehicle <NUM> (<NUM>). The remote control platform <NUM> may operate or control the vehicle <NUM> based on the driving instructions. The driving instructions, such as an increase in speed, turning, braking or other driving instruction converts to or is associated with an operation of the vehicle <NUM>. The remote control system <NUM> may engage or disengage the accelerator <NUM> and/or the brakes <NUM> to control the amount of speed, acceleration and/or deceleration of the vehicle <NUM>. Moreover, the remote control platform <NUM> may steer the vehicle <NUM>, such as to turn, using the steering <NUM>. The remote control platform <NUM> may change gears, such as operate the vehicle <NUM> in the driving mode, reverse mode or neutral mode. The remote control platform <NUM> may also change an orientation of the vehicle <NUM>, such as to turn the vehicle <NUM> degrees to drive the route in reverse while also avoiding the one or more objects. Other operations of the vehicle <NUM>, which may be controlled, include the control of the discharge of the electrical energy from the battery <NUM> to the motor and/or generator <NUM> and/or the state of the engine <NUM> or the throttle to move the wheels of the vehicle <NUM>.

In some implementations, the remote control platform <NUM> may operate other functionalities of the vehicle <NUM>, such as the windshield wipers, headlights, the windows, the heating, ventilation and air conditioning (HVAC) unit or other functions of the vehicle <NUM>. For example, the remote control platform <NUM> may turn on the seat warmers or the HVAC unit on a cold day to make the vehicle <NUM> more comfortable when the vehicle <NUM> arrives at the destination location of the user.

When the one or more objects do present a hazard to the vehicle <NUM>, the remote control platform <NUM> provides a notification to the user device <NUM> (<NUM>). The notification may indicate to the user that there is a hazard in the route of the vehicle <NUM> and request additional driving instructions. The notification may indicate the cancellation of the navigation of the vehicle <NUM> and/or provide the image data of the surrounding environment to allow the user to navigate around the one or more objects that present a hazard and/or to identify the hazard. The notification may include a suggestion of an alternative route to navigate the vehicle <NUM> around the one or more objects that present the hazard to the vehicle <NUM>. The alternative route may include sensor data of the relative position or location of the one or more objects to assist the navigation of the vehicle <NUM> around the one or more objects.

In some implementations, the remote control platform <NUM> may provide the image data along with the notification to the user device <NUM>. The image data may include live or real-time images of the surrounding environment from the one or more cameras 118a. The remote control platform <NUM> may provide an option for the user to take manual control over the vehicle <NUM>, which when activated, allows the user to navigate or control the operations of the vehicle <NUM> manually using the live or real-time images captured by the one or more cameras 118a.

The remote control platform <NUM> may cancel the navigation of the vehicle <NUM> when the hazard is detected (<NUM>). The remote control platform <NUM> may override any commands to the one or more vehicle components to operate, move or otherwise control the vehicle <NUM> to stop the vehicle <NUM> and/or to shutoff the vehicle <NUM> to prevent the vehicle <NUM> from moving. For example, the remote control platform <NUM> may shutoff the discharge of electrical energy from the battery <NUM>, turn off the engine <NUM>, apply the brakes <NUM> or otherwise disable operation of the vehicle <NUM> to prevent the vehicle <NUM> from moving. The cancellation may be automatic upon detection of the hazard within the route of the vehicle <NUM>.

In some implementations, the remote control platform <NUM> may receive a user command that automatically cancels the navigation or control of the vehicle <NUM> regardless of whether there is a hazard that is detected by the one or more sensors <NUM>. For example, the user may simply click on a button (e.g., an emergency or safety stop button) on the display of the user device <NUM> that causes the remote control platform <NUM> to turn off, immediately stop or otherwise immobilize the vehicle <NUM>. This advantageously allows the user to stop the vehicle <NUM> even when there are no hazards present and/or acts as an added safety feature.

<FIG> is a flow diagram of a process <NUM> for providing driving instructions to navigate the vehicle <NUM>. One or more computers or one or more data processing apparatuses, for example, the processor <NUM> of the user device <NUM> of the remote control system <NUM> of <FIG>, appropriately programmed, may implement the process <NUM>.

The user device <NUM> obtains the one or more locations of the vehicle <NUM> and/or the objects in the surrounding environment and the categorization of the one or more objects (<NUM>). The user device <NUM> may obtain the one or more locations of the vehicle <NUM> and/or the objects along with the categorization of the one or more objects from the remote control platform <NUM> and use this information to generate a rendering of the environment on the user interface <NUM>, such as a display, of the user device <NUM>.

The user device <NUM> generates icons for the vehicle <NUM> and the one or more objects (<NUM>). The generation of the icons may be based on the corresponding category of the one or more objects. For example, a tree icon <NUM> may be generated to represent a tree, another vehicle icon <NUM> may be generated to represent other vehicles, and a vehicle icon <NUM> may be generated to represent the user's own vehicle. The user device <NUM> may use a variety of other icons to represent different structures or other objects one may encounter in the environment surrounding the vehicle <NUM>.

Once the icons are generated, the user device <NUM> may render the icons on the display of the user device <NUM> (<NUM>). The distance, direction and location of the icons of the one or more objects relative to the icon of the vehicle <NUM> may mirror or mimic the actual distance, direction and location of the one or more objects relative to the vehicle <NUM> such that even the distance apart between the icons corresponds to or is correlated with the distance apart between the one or more objects and the vehicle <NUM>. This recreates an accurate rendering of the environment surrounding the vehicle <NUM>.

The user device may receive user input, such as a tap, a selection, a swipe or other user contact with the user interface <NUM> (<NUM>). The user input may correspond to driving instructions to control the vehicle <NUM>. The user device <NUM> may receive the user input via the user interface <NUM>, such as via a touch-screen display. Once the user input is received, the user device <NUM> may generate the driving instructions based on the user input (<NUM>). In some implementations, the remote control platform <NUM> may translate the user input into the driving instructions. <FIG> further describes the process <NUM> for associating user input with driving instructions.

The user device <NUM> may provide the driving instructions to the remote control platform <NUM> (<NUM>). The user device <NUM> may use the network access device <NUM> to send the driving instructions to the remote control platform <NUM> to allow the remote control platform to operate or control the one or more vehicle components to operate the vehicle <NUM>. The network access device <NUM> may send the driving instructions via the network <NUM>.

<FIG> is a flow diagram of a process <NUM> for generating the driving instructions to navigate the vehicle <NUM>. One or more computers or one or more data processing apparatuses, for example, the processor <NUM> of the user device <NUM> of the remote control system <NUM> of <FIG>, appropriately programmed, may implement the process <NUM>.

The user device <NUM> may receive the selection of an icon, such as the vehicle icon <NUM>, at a first location <NUM> (<NUM>). The selection may be of the vehicle icon <NUM>, as shown in <FIG> for example, and may indicate that the user intends to generate driving instructions for the vehicle <NUM>. The selection may occur when the user device <NUM> detects when pressure is applied to the display at the first location <NUM>, for example.

The vehicle icon <NUM> may be at a first location <NUM> on the display of the user device <NUM> surrounded by other icons, such as the other vehicle icon <NUM> and/or the tree icon <NUM>. The other icons may be positioned on the display in a position or location relative to the first location <NUM> that represents the starting location of the vehicle <NUM>. The first location <NUM> may be a starting location or a current location of the vehicle <NUM>. The starting location may be the beginning of the route of the vehicle <NUM>.

The selection of the icon may also indicate the orientation of the vehicle <NUM>. For example, when the user selects the vehicle icon <NUM>, the vehicle icon <NUM> may switch between different orientations of the vehicle <NUM> in which the vehicle <NUM> will travel, such as in a forward direction resulting in the vehicle <NUM> driving forward, or a rearward direction resulting in the vehicle <NUM> driving in reverse.

The user device <NUM> may obtain the movement of the icon to a second location (<NUM>). The user device <NUM> may receive user input that indicates that the user is dragging, sliding or otherwise moving the vehicle icon <NUM> from the first location <NUM> to a second location <NUM>. The dragging, sliding or otherwise moving of the vehicle icon <NUM> may indicate the route to navigate the vehicle <NUM> around the one or more objects, such as the other vehicles represented by the other vehicle icon <NUM>. The dragging, sliding or otherwise moving of the vehicle icon <NUM> around the one or more other icons, such as the other vehicle icon <NUM> and/or the tree icon <NUM>, may be represented by the path <NUM> on the display on the user interface <NUM>. The distance between the path <NUM> and the other vehicle icon <NUM> or tree icon <NUM> may correspond to the distance that the vehicle <NUM> maintains from the other vehicle represented by the other vehicle icon <NUM> or tree represented by the tree icon <NUM>.

The user device <NUM> may detect a release of the icon at the second location <NUM> (<NUM>). The second location <NUM> may be a destination location and the release of the icon may represent the end of the route at the destination location. The user device <NUM> may detect the release of the icon when pressure is released from the display at the second location <NUM>, for example. The release of the icon may trigger the generation of the driving instructions based on the user input sequence of the selection, dragging and subsequent release of pressure on the user interface <NUM>.

Once the user device <NUM> detects the release of the icon, the user device <NUM> may calculate or measure the speed of the movement along the path <NUM> from the first location <NUM> to the second location <NUM> (<NUM>). The calculated or measured speed of the movement of the user input may correspond to and indicate the speed that vehicle <NUM> is to travel on the route. The measured speed may be directly proportional to the speed that the vehicle <NUM> is to travel on the route around the one or more objects. As the movement speed of the pressure applied on the user device <NUM> increases, the speed that the vehicle <NUM> may increase, and as the movement of speed of the pressure applied on the user device <NUM> decreases, the speed that the vehicle <NUM> travels may also decrease.

The user device <NUM> may determine the relative distance between the path <NUM> of movement of the vehicle icon <NUM> and the other icons (<NUM>). The user device <NUM> may calculate the relative distance between the path <NUM> and the location of the other icons. The relative distance may correspond to relative distances that the vehicle <NUM> is to maintain with respect to the one or more objects.

The user device <NUM> generates the driving instructions (<NUM>). The user device <NUM> generates the driving instructions based on the determined relative distance between the path <NUM> and the other icons, the measured speed of the movement along the path <NUM>, the user input that indicates the selection of the icon at the first location <NUM> and the release of the icon at the second location <NUM>. The user device <NUM> may correspond the first location <NUM> to the start location of the vehicle <NUM>, the second location <NUM> to the destination location of the vehicle <NUM>, the path <NUM> with the route, the measured speed of the movement along the path <NUM> to the speed of the vehicle, and the relative distances between the path <NUM> and the other icons as the relative distance between the route of the vehicle <NUM> and the locations of the one or more objects. These driving instructions are then provided to the remote control platform <NUM> to operate or control the vehicle <NUM>.

Claim 1:
A remote control platform (<NUM>) for a vehicle (<NUM>), comprising:
a sensor (<NUM>) configured to detect one or more objects within a surrounding environment of the vehicle; and
an electronic control unit (<NUM>) coupled to the sensor and configured to:
receive, from a user device (<NUM>) wirelessly connected to the electronic control unit, user input data corresponding to a route between a starting location of the vehicle and a destination location within the surrounding environment, the user input data including a measurement of a speed at which a user's finger was dragged on the user device to create the route,
determine a vehicle speed at which the vehicle is to travel on the route based on the measurement being proportional to the vehicle speed,
determine an orientation of the vehicle and a direction of travel based on the user input data, and
navigate the vehicle at the determined vehicle speed along the route based on the one or more objects, the direction of travel and the orientation of the vehicle.