Source: https://patents.justia.com/patent/10092793
Timestamp: 2019-05-27 01:16:02
Document Index: 533647847

Matched Legal Cases: ['§ 119', 'Application No. 60', '§ 120', '§ 120', '§ 119', 'arty\n20140180451']

US Patent for Trajectory detection and feedback systems for tennis Patent (Patent # 10,092,793 issued October 9, 2018) - Justia Patents Search
Justia Patents With Electrically Actuated Or Operated IndicatorUS Patent for Trajectory detection and feedback systems for tennis Patent (Patent # 10,092,793)
May 30, 2017 - Pillar Vision, Inc.
This application is a continuation of U.S. patent application Ser. No. 13/745,429, filed Jan. 18, 2013, which is a continuation of and claims priority to U.S. patent application Ser. No. 12/015,445, filed Jan. 16, 2008. U.S. patent application Ser. No. 12/015,445 claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 60/880,773, filed on Jan. 16, 2007. U.S. patent application Ser. No. 12/015,445 further claims priority under 35 U.S.C. § 120 and is a Continuation-in-Part application of U.S. patent application Ser. No. 11/508,004, filed Aug. 21, 2006, now U.S. Pat. No. 7,854,669, which is a Continuation-in-Part and claims priority under 35 U.S.C. § 120 to U.S. patent application Ser. No. 10/242,373, filed Sep. 11, 2002, now U.S. Pat. No. 7,094,164. U.S. patent application Ser. No. 10/242,373 claims priority under 35 U.S.C. § 119(e) from the following three U.S. Provisional Patent Applications: 60/323,029, filed Sep. 12, 2001; 60/348,057, filed Jan. 11, 2002; and 60/395,875 filed Jul. 12, 2002. Each of the above provisional and non-provisional applications are incorporated herein by reference in their entirety.
The logic device may further designed or configured to determine the trajectory parameters associated with a two-dimensional trajectory for the tennis ball or to determine the trajectory parameters associated with a three-dimensional trajectory for the tennis ball. The logic device may be further designed or configured to determine one or more of a spin rate, a spin direction or combinations thereof of the tennis ball for at least one point along its trajectory. Further, the logic device is may be designed or configured to identify a racquet in the video frame data and to determine a position of the racquet, a velocity of the racquet, an orientation of the racquet as a function of time or combinations thereof as a function of time. Also, the logic device may be designed or configured to identify a body element of the player participating in the tennis related activity in the video frame data and to determine a position of the body element, an orientation of the body element, a velocity of the body element or combinations thereof, as a function of time and to provide feedback information related to one or more of the position of the body element, the orientation of the body element or the velocity of the body element to a user.
FIG. 1 is a diagram illustrating an in-situ use of trajectory detection and analysis system for tennis. In the FIG. 1, an analysis and feedback device and associated system 100 for analyzing a trajectory associated with the play of tennis and providing feedback may comprise one or more of the following: 1) one or more cameras (or 3D capturing sensors, such as CanestaVision™ Camera Module, Canesta, Inc., Americas Headquarters, Sunnyvale, Calif.) for recording video frame data used to characterize a trajectory (e.g., trajectory of a ball 110 along various points in its trajectory, such as at 110a, 110b, 110c and 110d), 2) a logic device (see FIGS. 6, 7A-C and 8 for more details and 3) and one or more feedback output mechanisms and associated interfaces for providing the feedback information to a user of the system (e.g., wearable device 122, cell phone 123 and wireless interface 100a). In some embodiments, the analysis feedback device may include sensors and/or an interface for measuring or inputting ambient conditions, such as temperature, humidity and wind speed.
Some examples of trajectories captured and analyzed by the device 100, may include a trajectory 102 of a tennis ball 110 hit by a racquet 108 swung by a player 116. In further detail, the device 100 may be designed or configured to capture, analyze and provide feedback associated with but not limited to: 1) a trajectory of a tennis ball after making contact with the court (trajectory after ball 110d hits court, which would be different for a hard court, grass court or clay court), 2) a trajectory of a body element of the human while playing tennis (such as a trajectory 104 of player's 116 hand swinging racquet 108 or trajectory a player's hand tossing ball 100), 3) a trajectory of a tennis ball tossed by the player 116 (e.g., along trajectory including 110a, 110b and 110c), 4) a trajectory 114 of a section of the racquet face or handle (which may provide, the changing angles of the racquet through the time of contact with the ball) and 5) a trajectory of a tennis ball 110 after being hit by the racquet 108, but before striking the ground, such as between trajectory points 110c and 110d.
The feedback information may be derived from measurements made by the analysis and feedback device 100. For instance, the feedback information may be related to one or more of the trajectory parameters associated with a tennis ball 110. The trajectory parameters, which may be determined by the analysis and feedback device 100 may include but are not limited to one or more of the following: 1) height of a serve toss 110b, 2) a lean of a serve toss, 3) a spin of a serve toss, 4) consistency of a serve toss and/or spin (derived from multiple serves), 5) an impact position height of a serve 110c, 6) an impact position lean of a serve, 7) a consistency of an impact position of a serve, 8) an initial speed, angle, direction and/or spin of a serve, 9) consistency of initial speed, angle, direction and/or spin of a serve, 10) a calculated speed of a serve, 11) a consistency of calculated speed of a serve, 12) calculated landing speed, location 110d, spin, and/or direction vector of a serve, 13) a consistency of calculated landing speed location, spin, and/or direction vector of a serve, 14) a measured landing speed, location, spin and/or direction vector of a serve, 15) a consistency of measured landing speed, location, spin and/or direction vector of a serve, 16) a height above/below the net at crossing point of the net and/or the location of the crossing point of the net, 17) release location of a serve toss, 18) combinations thereof. Further, the system may be able to record and provide feedback related to shot outcomes, such as in or out, a shot location on the court, hit the net, whether a shot was successfully returned, etc.
The feedback information provided to a player may be related to one or more parts of a trajectory. For instance, at one time a player may wish to receive feedback information regarding their toss 110a-110c, at another time a player may wish to receive feedback information regarding their serve speed and impact height (e.g., height at 110c), which corresponds to the trajectory of the ball after it is tossed, at other times the player may wish to receive feedback information regarding the location where the ball lands (e.g., 110d), which is the later in the trajectory 102. The analysis and feedback device 123 may be operable to provide feedback information for one or more parts of the trajectory simultaneously. For instance, the device may provide an audible, such as “7, 100, in,” which may correspond to a serve height of 7 feet, a serve speed of 100 miles per hour and an indication that it landed in. The analysis and feedback device may provide an interface that allows the user to select what type of feedback information they wish to receive, such as a combination of feedback parameters. In one embodiment, the feedback information may be provided to a wearable device 122 via wireless signals 101 from wireless interface 100a.
The analysis and feedback device may be operable to measure trajectory parameter at numerous locations along a trajectory, such as from toss, to impact, to landing and bounce as shown in FIG. 1 for a server. For example, the device may be able to measure the resulting location where the serve first landed 110d, such as in the opposite court. This location may be recorded simply as serve in or serve out, which may also be provided as feedback to the player. Further, the location could also designate the exact point that the serve hit the court or whether the serve hit a designated target space within the service box. It may or may not be necessary to add additional cameras to the device to achieve this measurement of resulting serve location. In one implementation, a second set of stereo cameras would be added to the device to view the court on the other side of the net from the server.
Additional details of an implementation of analysis and feedback device and/or system for tennis is described as follows for the purposes of explanation and is not meant to be limiting. FIG. 2 is a perspective view illustrating a use of a trajectory detection and analysis system for tennis. In FIG. 2, device 100 may be a custom tennis racquet bag containing a stereo camera consisting of top camera 124a and bottom camera 124b. The bag may also contain the logic device that receives the frames, generates the trajectory information and generates a wireless signal via wireless interface 100a with initial ball impact height, initial ball speed and initial ball angle. Wireless signal may be received by a wearable device 122, which may display feedback information, such as serve type, ball impact height, ball initial speed and ball initial angle (see FIG. 3 for more detail). The device may also be operable to output this information in an audio format. In some embodiments, the wearable device may also be used as an input device.
In particular embodiments, the wearable device may store information regarding a series of shots that can later be downloaded to a computer and can be uploaded onto the Internet, if desired. Thus, the device may include a memory unit, such as flash memory or a small hard-drive. In some embodiments, the device may include sensors, such as a 3-axis accelerometer and/or tilt/rotation sensors and/or GPS receiver that allow a position of the watch to be tracked. This information may be transmitted to the analysis and feedback device 100. In one embodiment, the watch may include a band or other surface that is coated with a material that allows it to be more easily tracked and discerned by cameras 124a and 124b.
The player 116a may select the service type to be attempted using the wearable device 122. The player may stand at the baseline in service position and hold the ball 122 in front of their body in the normal start position before the service toss. Both cameras may identify the ball allowing the logic device to calculate the distance from the bag to the ball. When the ball is identified, an LED light on the bag 100 may change from red to green signifying the device is ready for the player to hit the serve or another output device, such as an audio device may be used to indicate the device status. For instance, a speaker coupled to the bag. When the player serves, the cameras may track the trajectory of the ball during the toss, at impact point, and during the first 20 feet of the flight after racquet impact or at other points along the trajectory.
20 ft/170 mi/hr×1 mile/5280 ft×60 min/1 hr×60 sec/1 min×200 fr/1 sec=38.8frames
Of course, this calculation will vary depending on such factors as the camera being used, the player's physical attributes, the type of shot, etc. and is not meant to be limiting. Using a single camera, the impact height, initial serve speed and initial serve direction may be calculated in 2-dimensions. Further, using a stereo camera the analysis and feedback device may calculate true speed and direction in 3-dimensions correcting for the amount the serve direction deviates from parallel to the device. The stereo camera may improve ball detection by the vision system by separating the ball from the clutter of colors and patterns in the background. The device 100 may include an additional camera 124c for further tracking of the ball 108 along its trajectory.
In particular embodiments, the device 100 may be operable to track one or more of a series of shots after the serve or a series of shots in general. For instance, player 116a may be serving to a wall 126 or to another player (not shown). Player 116a (a refers to time a and position a) may hit the serve, which may return along trajectory 127 where 128b is the tennis ball along this trajectory. The player 116b (b refers to time b and position b). The player could then hit the ball against the wall or to another person, which could also be returned. The device 100 may be operable to record each of these shots and provide feedback information, which may vary or may be the same for each shot.
In one implementation, the wearable device 122 may be operable to store environment, trajectory and racquet data which can be uploaded to a computer or other device connected to a network, such as the Internet. The wearable device may also be operable to store video data, associated feedback information, analysis associated with a training session, such as consistency information. The wearable may also be operable to measure and store conditions associated with a training session, such as a player's heart and/or ambient temperature and humidity. Further, the device may be operable to communicate with an audio device, such as an ear-bud worn by the player using a protocol, such as Bluetooth.™
FIGS. 4 and 5 are examples of interface screens for a trajectory detection and analysis system for tennis. A display interface could appear as shown (In some embodiments, this display interface may be provided with the analysis and feedback device for immediate viewing or via another device, such as computer connected to the Internet for later viewing). On the interface screen, 150, “serve skill” may refer to the analysis done on the data and being presented. Further, the large frame in the screen includes the trajectory of the last 10 serves color-coded for toss height, a photo of player is in the background. “Serve Summary”, “Serve Log” and “Noah Rankings” are examples of other analyses/displays available that may be selected by a user. “Print Session” allows display page to be printed (or saved) to a file.“1st serve—flat” identifies the serve type. “10 serves” identifies the number of serves analyzed. Of course, as noted above, any type of stroke associated with the play of a game of tennis may be recorded and analyzed.
FIG. 7 is an information flow diagram for a trajectory detection and analysis system of the present invention. A sensor system 502, which may comprise emitters 506 and detectors 506, receives physical information 507. The physical information 507 may be energy signals reflected from a tracked object 508, such as a tennis ball. In the case where sensors are mounted to the tracked object 508, then the physical information 507 may be sent as signals from the sensors to a detector 504. Typically, the physical information 508 is transmitted through a medium such as air.
x ¨ = - ρ ⁢ ⁢ s 2 ⁢ ⁢ m ⁡ [ ( x . + w x ) 2 + y . 2 + ( z . + w z ) 2 ] ⁡ [ C D ⁢ cos ⁢ ⁢ α ⁢ ⁢ cos ⁢ ⁢ β + C L ⁢ sin ⁢ ⁢ α ⁢ ⁢ cos ⁢ ⁢ β ] y ¨ = - ρ ⁢ ⁢ s 2 ⁢ ⁢ m ⁡ [ ( x . + w x ) 2 + y . 2 + ( z . + w z ) 2 ] ⁡ [ C L ⁢ cos ⁢ ⁢ α ⁢ ⁢ cos ⁢ ⁢ β - C D ⁢ sin ⁢ ⁢ α ⁢ ⁢ cos ⁢ ⁢ β ] - g z ¨ = - ρ ⁢ ⁢ s 2 ⁢ ⁢ m ⁡ [ ( x . + w x ) 2 + y . 2 + ( z . + w z ) 2 ] ⁡ [ C D ⁢ sin ⁢ ⁢ β + C N ⁢ ⁢ cos ⁢ ⁢ α ]
{umlaut over (x)},ŷ,{circumflex over (z)} Acceleration components in x, y, z direction
{dot over (x)},{dot over (y)},ż Velocity components in x, y, z, direction
Racquet orientation, deg. 0 Initial speed, m/s 40 Headwind, m/s 0 Initial flight angle, deg. 0 Initial spin, rpm 0 Time step, sec .01 Air density, kg/m{circumflex over ( )}3 1.225 π 3.141593 Ball radius, m .033 Drag area, π * R2, m2 0.001327 Ball mass, kg 0.05 CD 0.63 CL 0.0 Initial inclination, rad 0.174533 Initial x-velocity, m/s 40.0 initial y-velocity, m/s 0.0 Gravitational acceleration, m/s 9.8 Initial vertical velocity, m/s 0.0 Initial horizontal velocity, m/s 40.0 Flight distance, m 25.0 Max height, m 9.0 Final speed, m/s 21.8 Final angle, deg. −17.2 Final horizontal velocity, m/s 20.8 Final vertical velocity, m/s −6.5
1. A device for analyzing trajectories of tennis balls, the device comprising:
one or more cameras for recording video frame data used to characterize a trajectory of a tennis ball generated by a player during a tennis related activity;
a logic device configured to i) receive the video frame data, ii) identify the tennis ball in the video frame data, iii) determine when a tennis racquet strikes the tennis ball, iv) generate trajectory parameters that characterize one or more states of the tennis ball along the trajectory based on the identified tennis ball in the video frame data, including at least one trajectory parameter measured by the logic device that characterizes a state of the tennis ball when the tennis racquet is determined by the logic device to strike the tennis ball, and iv) generate feedback information using the trajectory parameters, wherein the feedback information indicates the at least one trajectory parameter, and wherein the at least one trajectory parameter includes a height at which the tennis racquet strikes the tennis ball; and
at least one output mechanism for providing the feedback information.
2. The device of claim 1, wherein the tennis related activity is a toss of the tennis ball for a serve and wherein the logic device is further designed or configured to measure the trajectory of the tennis ball during the toss and prior to impact with a racquet.
3. The device of claim 1, wherein the feedback information includes one or more of the following 1) a height of a serve toss, 2) a lean of the serve toss, 3) a spin of the serve toss, 4) a consistency of a plurality of serve tosses, 5) a height above a net of the tennis ball, 6) an impact position lean of the serve, 7) a consistency of an impact position of a plurality of serves, 8) an initial speed of the serve, 9) an initial angle of the serve, 10) an initial direction of the serve, 11) an initial spin of the serve, 12) a consistency of the initial speed of the plurality of serves, 13) a consistency of the angle of serve of the plurality of serves, 14) a consistency of the direction of the plurality of serves, 15) a consistency of the spin of the plurality of serves, 16) a calculated speed of the serve, 17) a consistency of the calculated speed of the plurality of serves, 18) a calculated landing speed of the serve, 19) a location of the serve at landing, 20) a spin of the serve at landing, 21) a direction vector of the serve at landing, 22) a consistency of the calculated landing speed for the plurality of serves, 23) a consistency of the location at landing of the plurality of server, 24) a consistency of the spin at landing of the plurality of serves, 25) a consistency of the direction vector of the plurality of serves, 26) a measured landing speed of the serve, 27) a measured location of the serve, 28) a measured spin of the serve, 29) a measured direction vector of the serve, 30) a consistency of the measured landing speed of the plurality of serves, 31) a consistency of the measured location of the plurality of serves, 32) a consistency of the spin measured for the plurality of serves, 33) a consistency of the direction vector measured for the plurality of serves, 34) a height below the net of the tennis ball, 35) a location of the crossing point of the net of the tennis ball, 36) a release location of the serve toss and 37) combinations thereof.
4. The device of claim 1, wherein the logic device is further configured to identify a position of a body element of the player participating in the tennis related activity in the video frame data and to determine the position of the body element relative to a marking on the court.
5. The device of claim 1, wherein the logic device is further configured to determine whether a tennis ball hit by a player on a tennis court is inside of or outside of one or more boundary lines associated with the tennis court.
6. The device of claim 1, wherein the logic device is further configured to identify the tennis racquet in the video frame data and wherein the logic device is further configured to determine a position of the tennis racquet as a function of time, a velocity of the tennis racquet as a function of time, an orientation of the tennis racquet as a function of time or combinations thereof.
7. The device of claim 1, wherein the logic device is further configured to identify a body element of the player participating in the tennis related activity in the video frame data and wherein the logic device is further configured to determine a position of the body element, an orientation of the body element, a velocity of the body element or combinations thereof, as a function of time and generate feedback information related to one or more of the position of the body element, the orientation of the body element or the velocity of the body element.
8. The device of claim 1, wherein the logic device is further configured to determine, for the purposes of calibration, a distance from the device to one or more of the tennis ball, a racquet, a marking on a tennis court, a net on the tennis court, a vertical surface against which the tennis ball is being hit or a player hitting the tennis ball.
9. The device of claim 1, further comprising one or more sensors for determining an orientation of the device and wherein the one or more sensors comprise accelerometers or tilt sensors.
10. The device of claim 1, wherein the output mechanism includes a wireless interface for outputting the feedback information to one or more remote devices.
11. The device of claim 10, wherein the remote device is worn.
12. The device of claim 1, further comprising: a housing for the one or more cameras, the logic device, and the at least one output mechanism, the housing having a weight and form factor which facilitate one or more of transport, storage, unobtrusive set-up, calibration, or operation of the device.
13. The device of claim 1, further comprising an input mechanism.
14. The device of claim 13, wherein the input mechanism is a touch screen display.
15. The device of claim 13, wherein the input mechanism is a wireless interface for receiving input from a remote device.
16. The device of claim 1, wherein the logic device is a general purpose computer comprising: a processor, a data storage device, RAM, operating system software, device interfaces, device drivers and trajectory analysis software.
17. The device of claim 1, wherein the device is capable of one of autonomous set-up, autonomous calibration, autonomous operation or combinations thereof.
18. The device of claim 1, wherein after manual input of data by a user, a confirmation of data determined by the device, the logic device is further configured to complete a calibration procedure.
20. The device of claim 1, wherein the device is configured to determine for a plurality of related trajectories captured by the device a consistency for at least one of the trajectory parameters generated for each of the plurality of related trajectories.
21. The device of claim 20, wherein the consistency is determined by calculating a statistical deviation.
22. The device of claim 1, wherein the logic device is further configured to store data related to one or the trajectory of the tennis ball, movements of the player, movements of a racquet captured in the video frame data for use in a video simulation related to tennis.
23. The device of claim 1, wherein the logic device is further configured to identify a boundary line of a tennis court in the video frame data and to determine whether the tennis ball is inside or outside of the boundary line based on the identified tennis ball and the identified boundary line in the video frame data, and wherein the feedback information indicates whether the tennis ball is inside or outside of the boundary line.
24. The device of claim 1, wherein the logic device is configured to determine a height of a serve toss based on the identified tennis ball in the video frame data, and wherein the feedback information indicates the height of the serve toss.
25. The device of claim 1, wherein the logic device is configured to measure a spin of the tennis ball during a serve toss based on the identified tennis ball in the video frame data, and wherein the feedback information indicates the spin.
26. The device of claim 23, wherein the logic device is configured to determine a height of a serve toss based on the identified tennis ball in the video frame data, and wherein the feedback information indicates the height of the serve toss.
27. The device of claim 1, wherein the logic device is configured to calculate an average height at which the tennis racquet strikes a tennis ball for multiple tennis shots, and wherein the feedback information indicates the average height.
28. A method for analyzing trajectories of tennis balls, comprising:
recording, with one or more cameras, video frame data used to characterize a trajectory of a tennis ball generated by a player during a tennis related activity;
receiving the video frame data with a device having at least one processor for processing the video frame data;
identifying, with the device, the tennis ball in the video frame data;
determining, with the device, when a tennis racquet strikes the tennis ball;
generating, with the device, trajectory parameters that characterize one or more states of the tennis ball along the trajectory based on the identified tennis ball in the video frame data, wherein the generating comprises measuring, with the device, at least one trajectory parameter that characterizes a state of the tennis ball when the tennis racquet is determined to strike the tennis ball, and wherein the at least one trajectory parameter includes a height at which the tennis racquet strikes the tennis ball;
generating, with the device, feedback information using the trajectory parameters, wherein the feedback information indicates the at least one trajectory parameter; and
providing the feedback information with at least one output mechanism.
29. The method of claim 28, wherein the tennis related activity is a toss of the tennis ball for a serve, and wherein the method comprises measuring the trajectory of the tennis ball during the toss and prior to impact with the racquet.
30. The method of claim 28, further comprising determining, with the device for a plurality of related trajectories captured by the device, a consistency for at least one of the trajectory parameters generated for each of the plurality of relate trajectories.
31. The method of claim 28, further comprising determining, with the device, a height of a serve toss based on the identified tennis ball in the video frame data, wherein the feedback information indicates the height of the serve toss.
32. The method of claim 28, further comprising determining, with the device, a spin of the tennis ball during a serve toss based on the identified tennis ball in the video frame data, wherein the feedback information indicates the spin.
33. The method of claim 28, further comprising calculating, with the device, an average height at which the tennis racquet strikes a tennis ball for related tennis shots, wherein the feedback information indicates the average height.
4999603 March 12, 1991 Mele
5365427 November 15, 1994 Soignet
5430435 July 4, 1995 Hoch
5471383 November 28, 1995 Gobush
6671390 December 30, 2003 Barbour
20080200287 August 21, 2008 Marty
20140180451 June 26, 2014 Marty
Application Number: 15/608,490
Current U.S. Class: With Electrically Actuated Or Operated Indicator (273/371)
International Classification: A63F 9/24 (20060101); A63B 24/00 (20060101); A63B 69/38 (20060101); A63B 69/00 (20060101); A63B 69/36 (20060101); A63B 55/00 (20150101); G06K 9/00 (20060101); G06F 19/00 (20180101);