Self-stabilizing autonomous devices and use thereof in autonomous vehicles

Self-stabilizing autonomous devices and use thereof in autonomous vehicles are disclosed herein. An example device can include a body member having a central axis extending therethrough, a delivery platform disposed on a first end of the body member, a translation assembly disposed on a second end of the body member, a ballast assembly, and a controller that is configured to control operation of at least one of the translation assembly or the ballast assembly so as to balance forces exerted on the device and maintain the delivery platform in a desired orientation as the device moves across an operating surface.

FIELD OF THE DISCLOSURE

The disclosure generally relates to autonomous devices, and more particularly relates to devices, such as autonomous ballbots, which are capable of self-stabilization during operation and more specifically, but not by way of limitation, when operating in an autonomous vehicle such as a transit vehicle.

BACKGROUND

Autonomous devices are currently being deployed to provide various service-related functions. For example, autonomous devices can be used to transport riders, deliver inventory, and provide security monitoring—just to name a few. Moreover, while autonomous devices typically serve a general purpose and operate in a static environment, these autonomous devices are not well-suited to operating in a dynamic environment that exerts or causes exertion of forces on the autonomous devices that could destabilize the autonomous devices and cause them to tip over. For example, an autonomous device, which cannot self-stabilize, might likely tip over when operating inside a moving vehicle. Some or all of the above needs and/or problems may be addressed by certain embodiments disclosed herein.

DETAILED DESCRIPTION

Overview

The disclosure is related to self-stabilizing autonomous devices that operate in dynamic environments, such as within transit vehicles. In various embodiments, an example self-stabilizing autonomous device comprises a ballbot that incorporates a delivery platform. An example ballbot of the present disclosure is programmed to autonomously operate within a transit vehicle. In some embodiments, the ballbot facilitates commerce within the transit vehicle by autonomously delivering goods such as refreshments to passengers. In another embodiment, the ballbot can shuttle objects between passengers within the transit vehicle.

In some embodiments, the ballbot can also act like a conductor, collecting fares after the passenger sits down. If fares are collected at an entrance boarding/deboarding is too slow. If they are collected at a fixed transit stop, dynamic transit stops are infeasible. Thus, allowing an autonomous device such as a ballbot to perform these functions during transit solves commonly encountered issues during transit. In other non-limiting use cases, the ballbot can be configured to stow items like wet umbrellas or baggage from passengers assigned to a particular seat. In yet other embodiments the ballbot can function as a telepresence for a remote operator or even provide cleaning functions within the transit vehicle such as vacuuming and moping.

In various embodiments, an example ballbot of the present disclosure is configured with a ballast assembly and a translation assembly that can each be independently or cooperatively controlled using a controller in order to compensate for forces exerted on the ballbot due to a plurality of ballbot destabilizing conditions. In general, the plurality of ballbot destabilizing conditions comprise forces that are generated due to any combination of movement of the transit vehicle, movement of the ballbot, forces exerted on the ballbot by objects placed on a delivery platform of the ballbot, operating surface slope, and gravity. Taken together, the plurality of ballbot destabilizing conditions create forces (e.g., torques and/or heave) that can be measured relative to a contact point at which the translation assembly contacts the operating surface within the transit vehicle. In general, the operating surface is the floor of the transit vehicle on top of which the ballbot operates.

In one or more embodiments, part of the plurality of ballbot destabilizing conditions includes transit vehicle forces. An example ballbot of the present disclosure communicates with a controller or processor of the transit vehicle to receive the transit vehicle forces. These forces can include, but are not limited to, heave, pitch, roll, yaw rate, yaw acceleration, horizontal acceleration, and rotation of the transit vehicle.

The example ballbot of the present disclosure can process these transit vehicle forces in combination with input from an inertial measurement unit (IMU) of the ballbot that measures ballbot/device motion forces produced by movements of the ballbot as it moves around the transit vehicle on the operating surface. An example ballbot IMU is illustrated in the schematic diagram ofFIG. 3that is disclosed infra.

In some embodiments, the IMU of the ballbot will also utilize camera input obtained through a camera mounted on the ballbot. In various embodiments, the data sensed by the camera includes color images or video, range, or distance data. The camera input is processed to determine a slope of the operating surface, the movement of the ballbot on the operating surface, and the identification, location, and orientation of fiduciary objects located on the operating surface. In general, the fiduciary objects provide information to the ballbot that allow the ballbot to orient itself within the transit vehicle and to navigate within the transit vehicle.

In some embodiments, the ballbot can also utilize signals obtained from the transit vehicle that are indicative of motion such as steering, braking, velocity, suspension control, and the like. These signals provide the ballbot with advanced input that can be used to predict forces that are or will be exerted on the ballbot. For example, when the transit vehicle begins to turn, a signal indicating that the wheels are turning can be obtained from vehicle sensors that sense either movement of the steering linkages or movement within a steering gearbox. In another embodiment, rather than sensing specific movement, the vehicle sensors may sense an impending movement from other means such as a navigation system. For example, a sensor may indicate that the transit vehicle is within 100 feet of a right turn based on the current navigation plans of the transit vehicle. Other motions can be sensed through monitoring of other components of the transit vehicle that contribute to changes in transit vehicle motion.

In general, the ballbot can sense its orientation of the ballbot relative to vehicle coordinates when the ballbot is operating within a transit vehicle. The vehicle coordinates comprise three coordinate systems, road coordinates (generally a geodetic coordinate system such as longitude—latitude—elevation, and so forth). The vehicle coordinate system whose origin could be the (center of rotation) CR point projected to the floor of the vehicle and the ballbot coordinate system. Data (position, acceleration, mass, force, and so forth) used by the ballbot are expressed in one of the three and need to be translated between systems. Also, translations are constantly changing, requiring the ballbot to constantly adapt and stabilize during operation within the transit vehicle.

The vehicle coordinate system is the coordinate system rigidly connected to the operating surface the ballbot moves on. The vehicle suspension allows relative motion between the two coordinate systems. Road height sensors measure this motion and are provided to the ballbot over a wireless network.

In yet other embodiments, the ballbot can evaluate or consider an intended path of motion of the ballbot to predict possible instability. For example, if the ballbot is programmed to traverse a route within the transit vehicle, the ballbot can sense its current position and desired future position based on navigation instructions. This data can be used to predict or anticipate forces that will be experienced by the motion of the ballbot.

In view of the foregoing examples, a ballbot of the present disclosure is adapted to perform, if required, a multifactorial analysis considering each of the plurality of ballbot destabilizing conditions in order to compute forces that are acting on the ballbot. Generally, these forces function to destabilize or shift a center of mass of the ballbot, which results in instability of the ballbot. Correspondingly, when the ballbot is unstable, objects on the delivery platform on the ballbot may slide off. Thus, the ballbot of the present disclosure is configured to react to this center of mass shift by balancing the forces acting on the ballbot. In some embodiments, this balancing of forces is accomplished using a ballast assembly and/or movement of the ballbot through selective control. Broadly, the ballbot can self-stabilize in such a way that the delivery platform is held in a desired position (e.g., where objects on the delivery platform do not slide off).

In an example embodiment, a ballbot of the present disclosure is configured to move around the transit vehicle on the operating surface using the one or more fiduciary objects for navigation. The ballbot transports at least one object placed on a delivery platform, and the ballbot is configured to balance forces measured at a contact point between the ballbot and the operating surface. In some embodiments, the forces are caused by at least one of motion of the transit vehicle, motion of the ballbot, and/or gravity. In some embodiments, the ballbot balances the forces using at least one of a ballast assembly, a translation assembly, or a combination thereof. To be sure, the forces acting on the ballbot are balanced so as to stabilize the ballbot and to maintain a delivery platform of the ballbot in a desired orientation.

In some embodiments, the ballbots of the present disclosure can train in self-stabilization behaviors through motor babbling and/or iteratively comparing and copying the self-stabilization behaviors of other ballbots. When improvements in the self-stabilization behaviors are found, older and less advantageous self-stabilization behaviors are overwritten. This process can occur each time a ballbot of the present disclosure encounters another ballbot of the present disclosure.

Illustrative Architecture

Turning now to the drawings,FIG. 1depicts an illustrative architecture100in which techniques and structures for providing the systems and methods disclosed herein may be implemented. The illustrative architecture100may include a transit vehicle102and a ballbot104. A second ballbot105is illustrated, but is described in greater detail with respect to aspects disclosed in relation toFIG. 10. In one embodiment, the transit vehicle102comprises a transit vehicle controller106, an operating surface108, seats such as seat110, and a plurality of fiduciary objects, such as fiduciary object112. The transit vehicle102can comprise an autonomous vehicle in some embodiments. In other embodiments, the transit vehicle102is a non-autonomous (human-driven) vehicle.

The transit vehicle controller106can comprise a processor114, a memory116, and a wireless interface120. The transit vehicle controller106interacts with one or more sensors118. The processor114is configured to execute instructions stored in the memory116to navigate and drive the transit vehicle102. The processor114is also configured to execute instructions stored in the memory116to receive input from the one or more sensors118. In some embodiments, the one or more sensors118can include any combination of sensors that measure forces that are indicative of any of heave, pitch, roll, yaw rate, yaw acceleration, horizontal acceleration, and rotation of the transit vehicle. Each of these forces is disclosed in greater detail infra. In one example, a sensor can include a transit vehicle IMU that can comprise any of a three-axis accelerometer, a three-axis gyroscope, a three-axis magnetometer, and an altimeter.

These forces are generally referred to collectively as transit vehicle forces. The transit vehicle forces are transmitted to the ballbot104in some embodiments over the wireless interface120using a network122. The network122may include any one or a combination of multiple different types of networks, such as cable networks, the Internet, wireless networks, and other private and/or public networks. In some instances, the network122may include Bluetooth, cellular, near-field communication (NFC), Wi-Fi, or Wi-Fi direct. In some embodiments, the network122includes a device-to-device communication over a short-range wireless connection. When more than one ballbot is present in the transit vehicle102, the network122can include a mesh network. Thus, when the ballbot104and another ballbot105are operating in the transit vehicle102at the same time, the ballbots104and105can create a mesh network with the transit vehicle controller106in order to communicate with one another.

In one or more embodiments, the transit vehicle102can also include a ballbot console124that allows for recharging the ballbot104and for dispensing objects for sale and delivery using the ballbot104. For example, an object126such as a coffee cup can be obtained from the ballbot console124by the ballbot104. In an example use case, a passenger128can order the object126using an application on the passenger's mobile device130(e.g., smartphone). The ballbot console124receives the order and hails the ballbot104to pick up and deliver the object126to the passenger128.

In order to direct navigation of the ballbot104, the fiduciary objects such as fiduciary object112are positioned in various locations on the operating surface108. Each of the fiduciary objects comprises, for example, a means for encoding instructions that are utilized by the ballbot104. The instructions can comprise two- or three-dimensional objects such as quick response (QR) codes that encode information.

In one example, the fiduciary object112is a three-dimensional black and white QR code with information about the fiduciary object's location relative to the operating surface108. More information can be stored in a fiduciary object of a given size if color is also used to store coded binary information. If parts of the fiduciary object have different depths, the depths can also be used to code information. Thus, in some embodiments, the fiduciary object112comprises pixels that can each have a unique hue. The pixels can be read by the ballbot104as a sequence of color images where each pixel of the image is coded with the range (e.g., distance) between the pixel and the object. As noted above, the fiduciary objects are also oriented to give the ballbot104directional information as well as translation information. For example, the fiduciary object112is oriented to point towards another fiduciary object113. The ballbot104can sense or read the fiduciary object112and decipher its orientation in order to move itself to the location of the fiduciary object113if a desired destination for the ballbot104is the location of the fiduciary object113or another location that is navigated to via the ballbot104passing by or through the location of the fiduciary object113.

In some embodiments the camera captures three-dimensional images and uses these three-dimensional images to measure a distance between the camera and the pixel of a fiduciary object. For interpreting (for example) a QR code the processor of the camera uses an absolute distance to each pixel to determine the relative height of the pixels. The relative heights form a code that augments the code embedded in the pixel colors. Various three-dimensional camera technologies can be implemented, where some measure relative heights of the pixels without measuring the absolute distance. Three-dimensional cameras can sometimes replace LIDAR at lower cost. The aspects of both depth and color are included for completeness. Color three-dimensional QR codes can be printed with three-dimensional printers and can form a fiduciary. The orientation of the 3D code in the vehicle can help to orient the robot in the vehicles. QR codes are refereed for descriptive purposes, but any code using pixel color and height, and has an indication of direction can be used.

In various embodiments, the ballbot104can be configured to traverse between the fiduciary objects in a predetermined pattern such as the path132. In other embodiments, the ballbot104can directly navigate to a specific fiduciary object using other fiduciary objects as waypoints or guidelines to navigate to the desired fiduciary object. In some embodiments, each seat on the transit vehicle102is proximate to a fiduciary object so that passengers can obtain objects from the ballbot104without being required to leave their seats.

FIG. 2Adepicts in more detail the ballbot (e.g., device)104ofFIG. 1. The ballbot104generally comprises a delivery platform134, a ballast assembly136, a body member138, and a translation assembly140. In some embodiments, the body member138is a rigid tubular shaft having a first end142and a second end144. The delivery platform134can be coupled to the first end142of the body member138. The translation assembly140is coupled to the second end144of the body member138. A central axis CAextends through the body member138and is utilized as a reference for orientation of the ballbot104relative to the operating surface of the transit vehicle, as well as orienting the placement of components of the ballbot104. For example, the delivery platform134is positioned orthogonally to the central axis CA.

A stress sensor146is positioned at a coupling location between the delivery platform134and the first end142of the body member138. The coupling location can include any fixed or releasable connection or interface that joins the delivery platform134to the body member138. The stress sensor146senses any of shear force, normal force, and/or torque force caused by a mass of the delivery platform134and one or more objects on the delivery platform134such as the object126. Collectively, these forces are referred to in some embodiments as delivery platform force components. In general, the delivery platform force components can contribute to ballbot destabilization. For example, when the object126is placed near an edge of the delivery platform134, a weight of the object126induces a torque force that is measurable from both the interface that joins the delivery platform134to the body member138and a contact point where the translation assembly140contacts the operating surface108of the transit vehicle102.

In some embodiments, the delivery platform134can pivot at its coupling location between the delivery platform134and the first end142of the body member138. An actuator can tilt the delivery platform134if self-stabilization of the ballbot104is not completely achieved.

In some embodiments, the ballbot104comprises a ballbot vision system148. The ballbot vision system148can comprise a color video camera151and a processor152. Again, as noted above, the color video camera151comprises a three-dimensional color video camera used to detect the fiduciaries. When the video camera151images the floor of the vehicle the processor152can determine a gradient of the floor in the coordinate system of the robot. This data is input to the ballbot controller which moves the ball to maintain ballbot stability.

The processor152is configured to utilize the color video camera151to enable ground tracking and range finding. In certain embodiments, the processor152is configured to calculate slope of the operating surface108of the transit vehicle102. The processor152is also configured to calculate movement of the translation assembly140along the operating surface108of the transit vehicle102. Also, as noted above, the processor152can read data encoded into the fiduciary objects installed on the operating surface108of the transit vehicle102. In more detail, the video obtained from the color video camera151includes images of the fiduciary objects (QR code). The processor152utilizes QR code reading logic to read data encoded into the fiduciary object. Again, this information can include an identification of the fiduciary object (such as an identifier that uniquely identifies a fiduciary object), a location of the fiduciary object relative to the operating surface108of the transit vehicle102, and an orientation of the fiduciary object. This data enables the ballbot104to navigate around the operating surface108using the fiduciary objects as waypoints and/or navigation data sources.

With respect to motion of the ballbot104, the translation assembly140generally comprises a shell154and a sphere156. The shell154comprises a spherical sidewall158that at least partially encloses and retains the sphere156. The shell154can rotate clockwise and counterclockwise by way of a shell motor160(e.g., an actuator) that is located within the body member138of the ballbot104. In various embodiments, the sphere156can be coupled with the shell154using a retaining means159, such as a strap that both prevents selective disengagement of the sphere156from the shell154and allows the shell154to control directional turning of the sphere156. In this configuration, the sphere156is capable of both forward and rearward rotational movement that is orthogonal to the turning direction of the shell154.

The sphere156can be manufactured from a resilient material such as a rubber or plastic/polymer. In one example, the sphere156is an inflatable elastomeric ball. In various embodiments, the sphere156is rotated using sphere drives, such as a sphere drive163(e.g., omni-wheels placed perpendicular to an outermost surface of the sphere156). The number of sphere drives utilized in the ballbot104can vary according to design requirements. Also, the sphere drives can be actuated using output from the shell motor160. In some embodiments, idler balls mounted on a ring may be used to secure the sphere156to the shell154.

In one or more embodiments, the shell motor160and the sphere drive163are independently controllable by a ballbot controller162. Details regarding the ballbot controller162are disclosed with reference toFIG. 3. In general, the ballbot controller162selectively controls operations of the shell motor160and the sphere drive163(or motors) in order to move the ballbot104along the operating surface108of the transit vehicle102.

As noted throughout, the ballbot104is configured to self-stabilize due to a variety of forces that may be exerted against the ballbot104. These forces can be generated through movement of the ballbot104in a static environment. An example of a static environment could include a floor in a building. In these instances, forces are produced from movement of the ballbot104and placement of objects onto the delivery platform134of the ballbot104. Gravity is also acting on the ballbot104and can enhance the forces acting upon the ballbot104from its movement or loads the ballbot104carries.

To be sure, movement of the ballbot104induces a change in a center of mass MCof the ballbot104. When the ballbot104is in a stabilized configuration, the center of mass MCis vertically aligned with a contact point164, which is the point at which the sphere156contacts the operating surface108. When the center of mass MCshifts away from its vertically aligned, stabilized location, torque forces act upon the ballbot104, causing the ballbot104to tip. This tipping could result in the objects falling off of the delivery platform134or the ballbot104tipping over onto the operating surface108. When the center of mass MCshifts, gravity further pulls the center of mass MCdownwardly towards the operating surface108. As alluded to above, when the operating surface108is uneven and the ballbot104moves on a sloped portion of the operating surface108, this also induces shifting of the center of mass MCof the ballbot104. Thus, an additional destabilization component that can enhance forces exerted on the ballbot104includes a slope S of the operating surface108. It will be understood that the slope S of the operating surface108can include when the operating surface108is irregularly shaped rather than flat or when the operating surface108is angled due to a pitch of the transit vehicle102, as will be discussed in greater detail with reference toFIGS. 6A-6C.

In some embodiments, a shift in the center of mass MCof the ballbot104is a function of any combination of the transit vehicle force components, the delivery platform force components, the device motion forces due to motion of the device, the gravity force on the device, the transit vehicle slope, the slope of the operating surface, and the orientation of the device.

In some embodiments, a location of the center of mass MCof the ballbot104is determined by motor babbling. The center of mass MCof the ballbot104may change when objects are added or removed from the delivery platform134. Specific aspects of motor babbling would be known to one of ordinary skill in the art.

Broadly speaking, the ballbot controller162is configured to provide both stability and navigation signals for the ballbot104. In general, the ballbot controller162computes forces acting on the ballbot104at the contact point164and counteracts/balances the forces by selective operation of at least one of the ballast assembly136and/or the translation assembly140, or both in combination.

In some instances, such as when the operating surface108is flat, this balancing includes vertical realignment of the center of mass MCof the ballbot104and the contact point164of the sphere156. When the operating surface108has a slope S, the central axis CAof the ballbot104is positioned at an offset α relative to the contact point164of the sphere156.

Turning back to discussing slope computations, the ballbot104can use range-finding capabilities of the color video camera151of the ballbot vision system148and the processor152of the ballbot vision system148to detect a change in slope Δ of the operating surface108.

In more detail, an image of the operating surface108is sampled by the video camera151in pixels, each with its own range to a part of the operating surface108. The relative range of the pixels can be used to determine the slope of the operating surface108. Horizontal movement of the ballbot104with respect to the operating surface108can be determined using optical flow or phase-based motion detection—just as an example. Changing range for the operating surface108can also be tracked to determine if an angle between the ballbot104and the operating surface108is changing.

The slope S disclosed above relates specifically to surface irregularities of the operating surface108. Another type of slope, referred to as an operating surface gradient, can also be calculated. Specific details on how the ballbot104counteracts forces caused by changes in operating surface gradients are disclosed in greater detail with reference toFIGS. 7A-7C. Notwithstanding, one advantage of the self-stabilization aspects of the ballbot104is ensuring that the object(s) on the delivery platform134remain in contact with the delivery platform134during motion of the ballbot104and/or the transit vehicle102.

According to some embodiments, in addition to the forces disclosed above, other forces may be exerted upon the ballbot104when the ballbot104is operating in a dynamic environment, such as when the transit vehicle102is in operation. Thus, in addition to counteracting the forces on the ballbot104due to the slope S of the operating surface108, gravity, and the motion of the ballbot104, a set of forces applied to the ballbot104due to transit vehicle movement are also included in the stabilization computations performed by the ballbot controller162.

FIGS. 2B-2Ccollectively illustrate the ballast assembly136in greater detail. InFIG. 2B, the ballbot104is illustrated with the ballast assembly136in a neutral configuration.FIG. 2Cillustrates the ballast assembly136in an active position in order to counteract forces applied to the ballbot104.

In some embodiments, the ballast assembly136comprises a plurality of ballasts such as ballasts166A-166C. Additional or fewer ballasts than those shown are also likewise contemplated for use. In general, each of the ballasts such as ballast166A comprises a hub168, a body170, and a mass172. The hub168is both rotationally and hingedly coupled to the body member138of the ballbot104using an actuator174. Thus, each ballast has two possible ranges of motion. Ballast166A is illustrated to show a range of motion that includes a pivot or a hinge where an angle is created between the ballast166A and the body member138by movement of the ballast166A upwardly and outwardly from the body member138. The ballast166B illustrates the other potential range of motion, which includes a pendulum-type motion where a mass176rotates around a hub178that is connected to the body member138of the ballbot104using an actuator that is similar or identical to the actuator174of the ballast166A. In some instances, an actuator of a ballast can both rotate and hinge an associated mass at the same time. The movements of the ballasts166A-166C are coordinated and actuated by the ballbot controller162(seeFIG. 2A) to stabilize the ballbot104when destabilizing forces are acting against the ballbot104. As noted throughout, actuation of the ballast assembly136to shift ballast masses can be performed in combination with actuation of the translation assembly140(seeFIG. 2A) in some instances.

FIG. 3is a diagram of the sensing, control, and logic components of the ballbot104. The ballbot104comprises a ballbot IMU300, a ballbot vision system302(which corresponds to the ballbot vision system148ofFIG. 2A), and a ballbot controller304, which is identical to the ballbot controller162ofFIG. 2A, but is illustrated inFIG. 3schematically. A transit vehicle IMU306and a delivery surface stress sensor308(which corresponds to the stress sensor146ofFIG. 2A) each provide the ballbot controller304with signals indicative of forces produced by the transit vehicle and delivery platform, respectively. The ballbot controller304is configured to control the sphere drive(s)310and the ballast actuator(s)312to stabilize the ballbot104based on input received from the ballbot IMU300, the ballbot vision system302, and the delivery surface stress sensor308.

The ballbot IMU300can comprise a 3-axis accelerometer314, a 3-axis gyroscope316, a 3-axis magnetometer318, and an altimeter320. In general, the ballbot IMU300senses forces that act on the ballbot based on the motion of the ballbot. The ballbot IMU300can also integrate transit vehicle forces as well.

The ballbot vision system302corresponds to the ballbot vision system148ofFIG. 2A, and as with the ballbot vision system148comprises the camera151and the processor152. The processor152is generally configured to perform operations such as reading fiduciary objects, assessing operating surface slope, and calculating the motion of the ballbot based off of images and/or video obtained through the camera151. In some embodiments, the ballbot vision system302can comprise memory for storing executable instructions for performing any of these operations.

The ballbot controller304generally comprises a processor322and a memory324. In some embodiments, the memory324of the ballbot controller304stores a cognitive function and motion control module326and a stability and navigation module328. Cooperatively, the cognitive function and motion control module326and the stability and navigation module328include logic that can be executed by the processor322to sense destabilization of the ballbot104and to provide signals to the sphere drive(s)310and the ballast actuator(s)312in order to self-stabilize the ballbot104. Specific details on how the ballbot controller304senses destabilization of the ballbot104and compensates for the destabilization are disclosed in greater detail herein.

FIGS. 4-8Bcollectively illustrate transit vehicle forces that can contribute to destabilization of a ballbot of the present disclosure when the ballbot is operating in a transit vehicle. For example,FIG. 4illustrates various forces created by movement of a transit vehicle400. In general, each of these forces can be quantified as a change in acceleration in one or more directions (multi-directional changes in force are generally referred to as torque). The transit vehicle400can be subject to linear forces such as a linear longitudinal force402and a linear lateral force404. The linear longitudinal force402can act alone during forward movement of the transit vehicle400. A yaw force406is created when the transit vehicle400is turning and is a reaction to yaw acceleration of the vehicle. Additional details regarding the yaw force406are illustrated and described with reference toFIGS. 7A-7B. A heave force408is a lift force created when the transit vehicle400moves upwardly relative to the road410. In general, heave is a rate of change in a ride height of a transit vehicle averaged across all four wheels coupled with a rate of elevation change on a road. These forces are measured using a transit vehicle IMU (seeFIG. 1). Additional details regarding the heave force408are illustrated and described with reference toFIGS. 5A-5B.

In general, references to the term “linear” will be understood in context. Linear forces are approximately linear within a limited time interval. Therefore the logical model that controls ballbot stability is quasi-static, computing a ball movement based on current inputs and then recomputing as inputs change.

A pitch force412is a rotational force created when the transit vehicle400operates on a sloped surface, such as when the road410is sloped at an angle measured relative to the linear lateral force404. Additional details regarding the pitch force412are illustrated and described with reference toFIGS. 6A-6C. A roll force414is a rotational force created when the transit vehicle400operates on a sloped surface, such as when the road410is sloped at an angle measured relative to the linear longitudinal force402. The roll force414acts orthogonally to the pitch force412. To be sure, the roll force414and the pitch force412can act in combination when the road410is both sloped and pitched. Indeed, any combination (including all) of these transit vehicle forces can be generated at the same time based on the movement of the transit vehicle400.

FIGS. 5A and 5B, with reference toFIG. 3, collectively illustrate the behavior of an example transit vehicle500and a ballbot502operating inside of the transit vehicle500in response to an applied heave force. To be sure, positive heave is the component of movement in which all four wheels move up together, whereas negative heave is the component of movement in which all four wheels move down together. An area501that would create positive heave includes a section of a road505with a positive slope, whereas an area503that would create negative heave includes a section of the road505with a negative slope.

It will be understood that the heave of the transit vehicle500may cause a translation unit504of the ballbot502, and specifically its sphere506, to bounce and lose contact with an operating surface507of the transit vehicle500. This lack of contact may cause a loss in traction and stability when the heave switches from positive to negative. To counteract this problem, a future change in heave is predicted based on vehicle IMU input received by the ballbot502. In advance of the switch from positive to negative heave, the ballbot controller304activates actuators to lower the ballasts of a ballast assembly508and raises the ballasts when the ballbot IMU300(and specifically the 3-axis accelerometer thereof) on the ballbot502detects negative heave. Additional details on predicting vehicle heave are disclosed in U.S. Pat. No. 8,788,145, titled “Adaptive active suspension system with road preview” which is hereby incorporated by reference herein in its entirety including all references and appendices cited therein for all purposes. For purposes of clarity and brevity, the disclosure of the '145 patent is referred to herein as the “vehicle NVH model.”

In some embodiments, the stress sensor308of the ballbot502can detect a change in a normal force on the delivery platform510that is indicative of an arrival of a heave signal. As illustrated inFIG. 5B, and specifically in view514when the change in normal force is detected by the stress sensor308, the ballbot controller304actuates the ballasts (by hingedly raising the masses), which produces a down-thrust force that counteracts a change in heave force from positive to negative. Conversely, in view516, the ballbot controller304actuates the ballasts (by hingedly lowering the masses), which produces a upward-thrust force that counteracts a change in heave force from negative to positive.

FIGS. 6A-6C, with reference toFIG. 3, collectively illustrate pitch and roll forces which result in a change in gradient in the operating surface of a transit vehicle in which a ballbot is operating. This change in operating surface gradient is due to a transit vehicle602encountering road604slope/pitch changes such as driving on inclined roads. It will be understood that roll and pitch forces combine to produce an operating surface gradient. Broadly speaking, the operating surface gradient is a directional derivative measured between an operating surface600and a horizontal plane. The horizontal plane is roughly parallel with the road604when the road604is not inclined. A side view of the horizontal plane is illustrated inFIG. 6Aas a reference line HP. More specifically, the operating surface gradient SGis a function of an unloaded floor gradient of the transit vehicle602(which corresponds to the horizontal plane), a dynamic ride height RHof the transit vehicle602, coupled with changes in the slope of the road by the transit vehicle NVH model. The operating surface gradient SGcan be measured as the angle μ between the reference line HPand a current position of the operating surface600.

FIG. 6Billustrates a change in slope of the operating surface600due to a change in slope/pitch of a road604(seeFIG. 6A) between views606and608which causes a change in the operating surface600relative to the reference line HP. Generally, view606illustrates a large change in the operating surface gradient SG, and view608illustrates a large change in the operating surface gradient SGrelative to view606. In some embodiments, a large change in the operating surface gradient SGproduces a concomitant offset610between a contact point612of a sphere614of a translation unit616of a ballbot and an operating surface600. To be sure, the offset610can be measured as a distance between a central axis CAoperational of the ballbot and the contact point612of the sphere614.

FIG. 6Cis a top down view of the sphere614and the contact point612on the operating surface600illustrating an interplay between contact point offsets and floor gradients. A centerpoint CPof the sphere614is illustrated for reference. View618illustrates a stable condition for the ballbot. The contact point612is illustrated and a gradient line620is illustrated as extending between the centerpoint CPand the contact point612. For context, the gradient line620points downslope and its length conveys a steepness of the slope (e.g., magnitude of the gradient). In this illustration, the length is scaled such that the gradient line620reaches the centerpoint CPwhen the contact point612is stable. View624illustrates an unstable condition for the ballbot. An unstable condition occurs immediately after a floor gradient changes and before the ballbot controller304rotates the sphere614and moves one or more of the ballasts of a ballast assembly (not shown in this view, but can be seen inFIG. 2and/orFIGS. 5A-5B) to stabilize the ballbot. For illustrative purposes, a torque is created when a gradient line622is longer or shorter than a distance between the centerpoint CPand the contact point612.

In sum, to counteract changes in the offset due to changes in the floor gradient, such as a change in offset betweenFIG. 6Bviews606and608, the ballbot controller304can either move the sphere614or one or more of the ballasts of a ballast assembly to shift a center of gravity (seeFIG. 1) to counteract this change in contact point offset. The overall objective of the ballbot controller304is to balance all destabilizing forces acting at the contact point612of the ballbot.

FIGS. 7A-7B, with reference toFIG. 3, collectively illustrate forces such as yaw rate and yaw acceleration during vehicle rotation, their effects on a ballbot operating within the transit vehicle, and self-stabilization of the ballbot in response to these forces. An example transit vehicle700and ballbot704are illustrated inFIG. 7A. The transit vehicle700comprises an operating surface706and a center of rotation CR.

As noted above, a yaw force is derived from a rotational acceleration of the vehicle around a turning point. On front steer vehicles this point is midway between the two rear wheels. On four wheel steer vehicles it can change dynamically.

The ballbot controller304computes these accelerations and selectively controls any of a translation assembly708and/or a ballast assembly710of the ballbot704to balance or counteract these forces as illustrated inFIG. 7B.

In general, the transit vehicle700inFIG. 7Ais illustrated as performing a turning operation. Based on a location of the ballbot704on the operating surface706of the transit vehicle700, a lateral acceleration force is also exerted on the ballbot704. The centrifugal force is a function of a mass of the ballbot, a radius (distance between the ballbot704and the center of rotation CR), and a yaw rate of the transit vehicle700. To be sure, the yaw rate is a change in yaw acceleration that includes both linear712and longitudinal714acceleration components.

Views716-720respectively illustrate self-stabilization of the ballbot704in response to a change in yaw rate. In view716, the central axis CAof the ballbot704is in alignment with a reference line R that is substantially orthogonal to the operating surface706of the transit vehicle700. A yaw rate change is indicated by arrow722that acts on the ballbot704.

In view718, the ballbot controller304selectively controls operations of the translation assembly708and/or a ballast assembly710in view718to counteract a shift in the center of mass MCand the central axis CAof the ballbot704away from the reference line R. This shifting causes instability of the ballbot704. View720illustrates a return to vertical alignment of the center of mass MCand the central axis CAwith the reference line R, which is indicative of forces being balanced for the ballbot704and the ballbot704having self-stabilized.

FIGS. 8A-8B, in view ofFIG. 3, collectively illustrate horizontal acceleration during rotation of a transit vehicle800when turning, the effect of horizontal acceleration on a ballbot802operating within the transit vehicle800, and self-stabilization of the ballbot802in response to horizontal acceleration of the transit vehicle800.

It will be understood that horizontal acceleration804(shown as an arrow) is a function of lateral acceleration806and longitudinal acceleration808. To be sure, horizontal acceleration804is an inverse vector sum of both the lateral and longitudinal acceleration forces. The horizontal acceleration804will act against the ballbot802and will destabilize the ballbot802. As illustrated in views810-814ofFIG. 8B, the horizontal acceleration804of the transit vehicle800will act on a center of mass MCand create a torque force at a contact point816between the ballbot802and an operating surface818of the transit vehicle800. The ballbot802is in a destabilized position in view812.

Again, the horizontal acceleration804can either be sensed by the ballbot IMU300or can be received by the ballbot802from a transit vehicle IMU. In response, the ballbot controller304will selectively adjust either or both of a ballast assembly820and a translation assembly822in order to balance the torque at the contact point816and counteract any corresponding shift in the center of mass MCof the ballbot802. This force balancing returns the ballbot802to an upright and self-stabilized position as illustrated in view814.

As noted above, a location of the center of mass MCof the ballbot802is determined by motor babbling. The center of mass MCof the ballbot802may change when objects are added or removed from the delivery platform such as delivery platform824.

Taken together, the ballbot controller can be configured to receive any of the transit vehicle forces depicted and described inFIGS. 4-8B, can compute forces exerted at a contact point between a ballbot and an operating surface of the transit vehicle due to these transit vehicle forces, and can balance the transit vehicle forces in order to self-stabilize the ballbot.

To reduce complexity in ballbot controller computations, a ballbot controller can be configured to cause a ballbot to hold a current position when the ballbot itself senses or receives transit vehicle force information from a transit vehicle IMU that indicates a transit vehicle force is expected or is currently occurring. This simplifies calculations for the ballbot controller because no ballbot motion forces are produced by motion of the ballbot at the same time as the transit vehicle is in motion. The ballbot controller would permit movement once the transit vehicle forces cease. In some embodiments, the ballbot controller304can instantiate this pause or hold only when the transit vehicle forces meet or exceed (or are expected to exceed) a transit vehicle forces threshold(s).

Otherwise, in other embodiments, the ballbot controller allows the ballbot to move when the ballbot is experiencing transit vehicle forces. In these embodiments, the ballbot controller must not only account for changes in transit vehicle forces but also the relative motion of the ballbot as it traverses around the transit vehicle. The cooperative evaluation of both ballbot motion and transit vehicle is disclosed in greater detail with respect toFIG. 9.

Illustrative Processes

FIG. 9is a flowchart of an example process for effecting self-stabilization of a ballbot. The method is performed by the ballbot controller304(seeFIG. 3) and generally includes a step902of receiving operating surface slope data from a camera of a ballbot, as well as motion data of the ballbot relative to the operating surface. Using the slope and motion data of the ballbot relative to the operating surface, the method includes a step904of computing a location of a contact point of a ballbot on an operating surface.

In one or more embodiments, the method includes a step906of receiving transit vehicle forces from a vehicle IMU of a transit vehicle. Using the transit vehicle forces, the method includes a step908of determining a slope and/or pitch of the transit vehicle with respect to gravity. In some embodiments, the method includes a step910of determining a location of the ballbot within the transit vehicle, as well as a step912of receiving delivery platform force components such as shear, normal, and rotational forces caused by the delivery platform or loading of objects onto the delivery platform. The location of the ballbot within the transit vehicle can be used to calculate forces such as yaw rate, yaw acceleration, and horizontal force using the transit vehicle forces received in step908. Step914includes computing forces acting at the contact point of the ballbot due to any combination of vehicle motion and delivery platform loading.

Next, the method includes a step916of determining an intended motion for the ballbot with respect to a desired navigation path for the ballbot. In step918, the method includes determining ballbot orientation and motion forces from a ballbot IMU.

Step920utilizes the forces computed in step914caused by vehicle motion in combination with the ballbot orientation and motion forces calculated in step918and the intended motion for the ballbot in step916in order to compute movement of the translation assembly and/or ballast assembly of the ballbot which will reduce the forces acting on the ballbot to zero. Again, this can include compensating for a shift in the center of mass of a ballbot caused by the relative motion of the transit vehicle, the ballbot, and the delivery platform loading. When the desired movement of the translation assembly and/or the ballast assembly is determined, the method includes a step922of actuating movement of the translation assembly and/or the ballast assembly to complete the balancing of forces and stabilize the ballbot. The mechanisms for actuating the movement of the translation assembly and/or the ballast assembly are illustrated inFIG. 3.

Movement of the ballast assembly can comprise causing at least one of pivoting or hinging of one or more of a plurality of ballasts in order to balance the forces exerted on the device. Directional movement of the translation assembly, including rotating a shell and/or a sphere of the translation assembly can also be utilized to produce the desired balancing of forces that will stabilize the ballbot. Moreover, the translation assembly is used to move the ballbot through the transit vehicle. In some embodiments, this movement is accomplished by reading data encoded into the fiduciary objects for navigating the ballbot on the operating surface, as well as selectively controlling the translation assembly to perform the desired navigation.

With respect to navigation and intended movement of the ballbot, in some embodiments, the method includes a step of sensing any of identification, location, and orientation of fiduciary objects on the operating surface. This can occur using image processing of video/images captured by a camera mounted on the ballbot. In some embodiments, the ballbot navigates through the transit vehicle using the fiduciary objects. The ballbot can perform specific routing between adjacent fiduciary objects in some embodiments.

In one example, a first fiduciary object can encode data that instructs the ballbot with data that allows the ballbot to translate to a second, adjacent fiduciary object. In more detail, a ballbot controller is configured to utilize an intended motion of the ballbot during navigation and the orientation of the ballbot to further control the selective movement of one or more of the plurality of ballasts of a ballast assembly or directional movement of the translation assembly.

As noted above, the ballbot can receive transit vehicle force components from signals provided by vehicle force sensors, as well as suspension signals and steering signals of the transit vehicle, according to the vehicle NVH model referenced herein.

FIG. 10is a schematic diagram that illustrates two ballbots1002and1004, which are configured to iteratively learn self-stabilization behaviors from one another through imitation. Two example ballbots are illustrated inFIG. 1, such as ballbot104and ballbot105. Each of the ballbots1002and1004includes the components of the ballbot104ofFIG. 3, generally illustrated as shared ballbot components1001such as the ballbot IMU, the ballbot controller, the sensors, the actuators, and so forth.

While two ballbots are illustrated, it will be understood that any number of ballbots can learn self-stabilization behaviors from one another through imitation. Additionally, in order to implement imitation and self-stabilization behaviors, the ballbot1002can include a comparator1006and a switch1008that selectively choose between one of two possible self-stabilization behavior strategies1010and1012, respectively. The first self-stabilization behavior strategy1010includes a self-stabilization behavior received from the ballbot1004. The second self-stabilization behavior strategy1012includes a self-stabilization behavior that originates from the ballbot1002. The ballast assembly and/or translation assembly1014of the ballbot1002are controlled based on a selection made by the comparator1006. The selection includes a choice between self-stabilization behavior strategies1010and1012. In some embodiments, the ballbot1002can overwrite the self-stabilization behavior strategy1012with self-stabilization behavior strategy1010when the comparator1006determines that the self-stabilization behavior strategy1010is more effective than the self-stabilization behavior strategy1012.

Ballbot1004is likewise provided with a comparator1016and a switch1018that selectively choose between one of two possible self-stabilization behavior strategies1020and1022, respectively. The first self-stabilization behavior strategy1020includes a self-stabilization behavior received from the ballbot1002. The second self-stabilization behavior strategy1022includes a self-stabilization behavior that originates from the ballbot1004. The second ballbot105also includes a ballast assembly and/or translation assembly that is/are controlled using one of the selected self-stabilization behavior strategies.

Example Embodiments

In some instances, the following examples may be implemented together or separately by the systems and methods described herein.

Example 1 may include a device, comprising: a body member having a central axis extending therethrough, the body member having a first end and a second end; a delivery platform disposed on the first end of the body member and orthogonal to the central axis; a translation assembly disposed on the second end of the body member providing for movement of the device; a ballast assembly; and a controller that is configured to control operation of at least one of the translation assembly or the ballast assembly so as to balance forces exerted on the device and maintain the delivery platform in a desired orientation as the device moves across an operating surface.

Example 2 may include the device of example 1, further comprising one or more delivery platform sensors that are configured to measure delivery platform forces comprising at least one of shear, normal, or torque forces applied to the delivery platform by one or more objects placed onto the delivery platform.

Example 3 may include the device according to example 2 and/or some other example herein, further comprising a camera that is configured to capture camera input comprising at least one of: a slope of the operating surface that is contacted by the translation assembly; movement of the translation assembly relative to the operating surface; or identification, location, and orientation of fiduciary objects on the operating surface.

Example 4 may include the device according to example 3 and/or some other example herein, further comprising an inertial measurement unit configured to measure the forces exerted on the device at a contact point where the translation assembly contacts the operating surface, wherein the forces exerted on the device comprise device motion forces due to motion of the device and gravity force.

Example 5 may include the device according to example 4 and/or some other example herein, wherein the inertial measurement unit is further configured to: sense an orientation of the device relative to vehicle coordinates when the device is operating within a transit vehicle; and receive a transit vehicle slope from the transit vehicle.

Example 6 may include the device according to example 5 and/or some other example herein, wherein the inertial measurement unit is further configured to receive transit vehicle force components that are indicative of forces of the transit vehicle provided by vehicle force sensors, as well as suspension signals and steering signals.

Example 7 may include the device according to example 6 and/or some other example herein, wherein balancing forces exerted on the device comprise compensating for a shift in a center of mass of the device, wherein the shift in the center of mass is a function of: the transit vehicle force components; the delivery platform force components; the device motion forces due to motion of the device; the gravity force on the device; the transit vehicle slope; the slope of the operating surface; and the orientation of the device.

Example 8 may include the device according to example 7 and/or some other example herein, wherein the ballast assembly comprises a plurality of ballasts that are each rotatably and hingedly coupled to the body member, and provide movement in two axes of motion relative to the body member, wherein the controller causes at least one of selective movement of one or more of the plurality of ballasts or directional movement of the translation assembly in order to produce the compensation for the shift of the center of mass of the device.

Example 9 may include the device according to example 8 and/or some other example herein, wherein the controller is further configured to: read data encoded into the fiduciary objects for navigating the device on the operating surface; and selectively control the translation assembly to perform the navigation.

Example 10 may include the device according to example 9 and/or some other example herein, wherein the controller is further configured to utilize an intended motion of the device during navigation, and the orientation of the device is computed by the inertial measurement unit to further control the selective movement of one or more of the plurality of ballasts or directional movement of the translation assembly.

Example 11 may include the device according to example 1 and/or some other example herein, wherein the desired orientation includes the delivery platform held at a position that prevents objects contacting an upper surface of the delivery platform from sliding off of the delivery platform when the device is in motion.

Example 12 may include a method, comprising: determining, by one or more processors coupled to at least one memory, a location of a contact point between a translation assembly of a ballbot and an operating surface of a transit vehicle; sensing a slope and movement of the ballbot on the operating surface using camera input from a camera; receiving transit vehicle force components for the transit vehicle; computing a slope of the operating surface with respect to gravity; sensing a location of the ballbot within the transit vehicle; determining forces on the ballbot at the contact point comprising any of the transit vehicle force components, the location of the contact point, the slope, the movement of the ballbot, the slope of the operating surface with respect to gravity, and the location of the contact point of the ballbot, or any combination thereof; and selectively controlling at least one of a ballast assembly and a translation assembly of the ballbot so as to balance the forces on the ballbot at the contact point.

Example 13 may include the method according to example 12, further comprising measuring delivery platform force components comprising at least one of shear, normal, or torque forces of a delivery platform of the ballbot.

Example 14 may include the method according to example 13 and/or some other example herein, wherein the ballbot further senses identification, location, and orientation of fiduciary objects on the operating surface.

Example 15 may include the method according to example 14 and/or some other example herein, wherein the ballbot navigates through the transit vehicle using the fiduciary objects.

Example 16 may include the method according to example 15 and/or some other example herein, further comprising sensing: an orientation of the ballbot relative to vehicle coordinates when the device is operating within a transit vehicle; and gravity and motion forces created by movement of the device along the operating surface.

Example 17 may include the method according to example 16 and/or some other example herein, further comprising receiving the transit vehicle force components from signals provided by vehicle force sensors, as well as suspension signals and steering signals of the transit vehicle.

Example 18 may include the method according to example 17 and/or some other example herein, further comprising causing at least one of pivoting or hinging of one or more of a plurality of ballasts or directional movement of the translation assembly in order to balance the forces exerted on the device.

Example 19 may include the method according to example 18 and/or some other example herein, further comprising: reading data encoded into the fiduciary objects for navigating the ballbot on the operating surface; selectively controlling the translation assembly to perform the navigation; and wherein the controller is further configured to utilize an intended motion of the ballbot during navigation and the orientation of the ballbot to further control the selective movement of one or more of the plurality of ballasts or the directional movement of the translation assembly.

Example 20 may include a system, comprising: a transit vehicle comprising an operating surface having one or more fiduciary objects positioned thereon; and a ballbot configured to move around the transit vehicle on the operating surface using the one or more fiduciary objects for navigation, wherein the ballbot transports at least one object placed on a delivery platform, the ballbot being configured to balance forces measured at a contact point between the ballbot and the operating surface, wherein the forces are caused by at least one of motion of the transit vehicle, motion of the ballbot, or gravity, wherein the ballbot balances the forces using at least one of a ballast assembly of the ballbot, a translation assembly of the ballbot, or a combination thereof, the forces being balanced so as to stabilize the ballbot and to prevent the at least one object from sliding off the delivery platform.