Abstract:
A method and apparatus for providing realistic gun motion input to a video game. In one embodiment, a plastic enclosure houses a video camera, PCB, microcontroller processor, and many buttons for controlling various aspects of a video game. The processor examines images from the video camera using various feature tracking algorithms and determines the direction and magnitude of motion of the video camera, and hence the motion of plastic gun shaped enclosure in which the video camera is mounted, and that the game player is wielding. This motion data is translated into motion data that a video game running on a video game console can understand, and transmitted to the video game console. The end result is that a user pointing and moving the plastic gun will cause the in game character of the video game to move and point its gun in concert with the game player, thereby providing an intuitive and fun aiming mechanism for playing video games. The button presses that the game player initiates on the apparatus are also reported to the video game and also affect various actions therein.

Description:
BACKGROUND OF THE INVENTION  
       [0001]     This invention relates to the field of video games, specifically allowing a user to play a video game by moving and pressing buttons on a peripheral that is shaped like a gun.  
         [0002]     There is a class of video games known as “first person shooters”, abbreviated FPS. In these FPS games the video game screen generally shows the view point of a character and generally there is a cross-hair or other type of reticule to show where the character is currently looking and aiming their weapon.  
         [0003]     Since approximately the middle 1990&#39;s the preferred control mechanism for FPS games played on a personal computer (PC) has been using a mouse and a keyboard. The mouse is used to control the aiming and direction of view, which is usually indicated on screen by a small reticule, and the keyboard keys are used to make the character move forward, backward, left, right, and diagonal throughout 3D world.  
         [0004]     On video game consoles these FPS games are played with a gamepad type controller. These gamepad controllers are gripped with both hands and have two small joysticks mounted on the top side which are operated by the user&#39;s thumbs. These joysticks are typically referred to as “thumbsticks”. There are also several buttons located on various other locations of these gamepad type controllers.  
         [0005]     When playing an FPS game with a gamepad controller usually one of the thumbsticks is used for controlling the wind age and elevation of the in-game character&#39;s reticule, and therefore it controls the direction of view and aiming of the in-game character. The other thumbstick controls the character&#39;s movement throughout the 3D world in the forward, backward, left, right, and diagonal directions.  
         [0006]     Gamepad type controllers do not provide very precise control. With the gamepad type controllers that come with most video game consoles the aiming and movement are controlled with the user&#39;s thumbs using two thumbsticks. These thumbsticks usually are internally composed of two potentiometers: one to measure the X-axis motion of the thumbstick and one to measure the Y-axis motion of the thumbstick. Using these thumbsticks adversely affects aiming and movement precision in several ways. One of these ways is that controlling a thumbstick with a single thumb lacks the opposing forces granted by the use of multiple fingers applied to a single input mechanism. Another disadvantage, of this two axis per thumbstick configuration, is that there is an inherent motion bias due to friction along both of these axes. Controllers that offer equal freedom of movement in any given direction provide a more fluid, intuitive, and accurate input means.  
         [0007]     Furthermore the gamepads are unsatisfactory to many game players because one has to have two thumbs on the thumbsticks controlling the direction of view and movement, and the remaining fingers wrapped around the underside of the controller to grip it. This leaves no available fingers to push the buttons on the top side of the controller. This is especially problematic in FPS games because there are many important functions that are assigned to these top buttons that are needed while one&#39;s thumbs are busy aiming and moving. This leaves the gamepad user no other choice but to remove a thumb from a thumbstick to press a button when needed. This causes the user&#39;s in-game character to momentarily stop moving or aiming, thereby leaving the player&#39;s character more vulnerable to the hazards in the video game, such as enemies in the video game firing their weapons at the user&#39;s character.  
         [0008]     A disadvantage of the keyboard and mouse method of controlling video games is that it is a cumbersome method to play a video game in a living room setting. This is mainly because a smooth and flat surface is necessary for the mouse, and a sturdy surface is necessary for the keyboard. Both of these surfaces need to be in close proximity to one another and the game player, and positioned for good ergonomics. Such a configuration is unlikely to be found in most living room environments.  
         [0009]     Another disadvantage of the keyboard and mouse configuration is that left/right and forward/back movement within the video game is performed using keys on the keyboard. Since keys on a keyboard only have two states, on and off, the speed of movement cannot be modulated smoothly and precisely.  
         [0010]     Lastly, and most important, a disadvantage of both the gamepad and keyboard/mouse methods of controlling FPS games is that they are not realistic methods of aiming a gun.  
       BRIEF SUMMARY OF THE INVENTION  
       [0011]     The present invention is generally directed to a gun shaped peripheral that makes it possible for a game player to control FPS (and other) video games in a very natural, intuitive, precise, and fun manner. Several objects and advantages of the present invention include, but are not limited to:  
         [0012]     (a) to allow players to use a peripheral that is shaped like, and is held like, a real firearm;  
         [0013]     (b) to provide an intuitive method of aiming the in-game gun by simply aiming a gun shaped peripheral;  
         [0014]     (c) to allow keyboard/mouse players to play games in a more relaxed posture, rather than being bound to a desk;  
         [0015]     (d) to allow a player to fire the in-game gun by pressing a button that is actuated in a similar manner as the trigger on a real gun;  
         [0016]     (e) to provide well located buttons and actuators for controlling a video game on a gun shaped peripheral;  
         [0017]     (f) to provide a very immersive game playing experience.  
         [0018]     In one embodiment of the present invention, the gun shaped peripheral plugs into a video game console. The peripheral has a microprocessor which outputs signals to the video game console that are in the format of a standard gamepad type controller for that video game console. Before outputting the signals to the video game console, the user actions are processed in accordance with user defined settings and mappings. For instance, a user may decide that a button on the peripheral should produce the same effect as would pressing the “A” button of the standard controller for that video game console. This and other user defined information is stored in a non-volatile memory such that it will be stored even after the device is unplugged and without power. These settings can be configured by using a special button mapping button. This embodiment also has a thumbstick mounted on the pistol grip area, such that it is operated by the player&#39;s thumb.  
         [0019]     In another embodiment of the present invention the gun shaped peripheral communicates to the game system via a wireless connection. Also, the sensitivity of the motion of the gun shaped peripheral as reported to the video game system can be adjusted to suit the player&#39;s preference. The plastic enclosure has an expandable stock so that the user can brace it against their shoulder for added precision and realism. A solenoid is attached that produces a percussive kick back when the trigger is depressed, in order to simulate a rifle recoil force. Buttons are located on the fore grip and trigger area of the peripheral for easy access.  
         [0020]     Note that the invention is not limited to the aforementioned embodiments, but rather these are examples meant to help crystallize the invention in the mind of the reader. For instance, the device itself can be connected to the console via many different physical and wireless means. 
     
    
     BRIEF DESCRIPTION OF DRAWINGS  
       [0021]      FIG. 1  is a block diagram showing important elements and signal flow for a preferred embodiment of the invention. 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0022]     The present invention is related to an apparatus and method to allow a user to aim in a video game by holding and aiming a gun shaped peripheral. Referring to  FIG. 1 , in a first embodiment  100 , a video camera  110 , microcontroller processor  120 , fifteen buttons  115 , a video game controller plug  130 , an EEPROM memory chip  135 , and a thumbstick  105  are mounted to a PCB (Printed Circuit Board) which connects the aforementioned components via electrically conductive traces. The PCB board is mounted in a plastic enclosure that is shaped like a sub-machine gun. The camera  110  is mounted such that it is at the front of the apparatus and can take images of what is in front of the apparatus. When the video game controller plug  130  is plugged into the video game controller receptacle  145  of a video game console  140 , the electricity from the video game console  140  provides power to the components on the PCB of the apparatus. The camera  110  continually generates images of the environment in front of the apparatus. The images are read by the processor  120 . The processor  120  feeds the images to the image processor  125 . The image processor  125  examines the images in order to detect the direction and magnitude of the motion of the camera  110  and thus the direction and magnitude of the motion of the front end of the apparatus. The image processor  125  does this by identifying trackable features within the images, locating those features in subsequent images, and calculating the displacement of the features along the X and Y axes from one frame to the next. The displacement information is then translated by the processor  120  into the type of displacement information that the video game console  140  understands. The displacement information is optionally modified according to a sensitivity adjustment setting prior to sending it to the video game console  140 . The processor  120  then sends the displacement information to the video game console  140  via the video game controller plug  130  and video game controller receptacle  145 . The aforementioned sensitivity setting allows the user to configure the apparatus to produce larger displacement values for a given movement of the apparatus, or smaller displacement value for a given movement of the apparatus. The displacement values as modified by this sensitivity setting are sent to the video game console  140 . The displacement values are modified by the sensitivity setting by multiplying the displacement values by a sensitivity value. A sensitivity value of less than 1.0 will shrink the displacement value, and a sensitivity value of greater than 1.0 will amplify the displacement value, and therefore increase the distance the in game character move their weapon per unit of distance the game player moves the plastic gun.  
         [0023]     Features are selected by the image processor  125  that are highly trackable. A highly trackable feature is one that is significantly different from another image (of the same number of pixels) that is only a small distance away, for instance one pixel away. So to find a highly trackable feature, the pixels of the feature are compared with every set of pixels that is one pixel in distance away. If all sets of pixels are significantly different than the feature, then the feature is selected for tracking. Once a feature has been selected it is then searched for in subsequent images. A search is done in a subsequent image by comparing the feature to many sets of pixels that are within a certain pixel distance of where the feature was originally found. Because of computational limitations a full search for the feature is not done on the entire image. If the feature that is being tracked has moved outside of the search radius in a subsequent image, then a lower resolution version of the image can be searched for the same feature, or optionally a different feature. The advantage of using the lower resolution image is that it has far fewer pixels and therefore more of the image can be searched, and therefore features can be tracked when they are moving faster (objects moving faster generally require a larger search radius because they travel more distance between frames). The disadvantage of this method is that once the feature is found then its displacement can only be reported using the lower resolution granularity of the smaller resolution image. Therefore the precision of movement is diminished compared with the higher resolution image. However, using the rough displacement/location value that the lower resolution image has yielded for the feature, this provides a rough idea of where the feature is within the higher resolution image. Therefore the higher resolution image can be searched for within a much smaller search radius. This now makes searching the high resolution image computationally feasible. When combining the search of the low resolution image with the search of the high resolution image, the image processor  125  is not only able to track fast moving objects, but the image processor  125  is also able to track these objects with high precision as well.  
         [0024]     The processor  120  also reads the status of several buttons  115  that are attached to the PCB. The buttons  115  are for controlling various actions within a video game. The pressed or released status of all of the buttons  115  are read, and then a non-volatile EEPROM memory  135  is read to determine to which function of the video game each button is mapped. The status of these video game function values are then sent to the video game console  140  via the video game controller plug  130 . This mapping within the EEPROM memory  135  allows a game player to assign each button to a chosen function. Also, the game player can choose to save a collection of these button mappings to the EEPROM memory  135 . The game player can then cause the processor  120  to recall the collection of mappings and a later time. The user can store many such collections in the EEPROM memory  135 , and then instruct the apparatus to use a specific mapping set depending upon which game the game player has chosen to play. These mappings and other settings are stored in an area of the EEPROM memory  135  known as a profile. The user can recall the appropriate profile for a particular game at any time.  
         [0025]     The processor  120  also monitors the position of a thumbstick  105  that is attached to the PCB, and sends its value to the video game console  140  via the video game controller plug  130 . This thumbstick is used to control the forward, backward, left, and right movement of the in game character within the video game&#39;s 3D world.  
         [0026]     Also in this embodiment, one of the buttons  115  on the plastic enclosure is used by the game player to reposition the apparatus without causing motion in the video game. This is important because the game player could initially be aiming the apparatus in a comfortable direction and position. But then, if the user would like to cause the in game character to look very far to the right, then this might cause the game player to not be able to comfortably move the apparatus far enough to right to cause the in game character to move as far as needed. In this example, the user does not want to continue aiming further to the right, because they will no longer be in a comfortable position, and in fact this may cause them to turn facing away form the video game. But the game player wants to move the in game character further to the right, so what is needed is a button, that when pressed, will allow the apparatus to move in any direction without causing any corresponding motion of the character within the video game. The game player can press this button and then turn aim the gun further to the left, and the in game character will not adjust its aim at all. When the game player then releases this button, subsequently moving the apparatus to the right will cause the in game character to move to the right even further. In this manner, the game player is able to adjust their aim without causing unwanted move in the video game. This is similar in concept to what an FPS game player on a PC does when using a mouse. When the PC player wants to move the mouse without affecting the action on the screen, they simply lift the mouse off the table so that they can move it without causing any input to the PC. If they were not able to do this then they might find themselves in a situation where the mouse is, for example, moved to the far left edge of the desk, and therefore the in game character cannot be moved any further to the left. The processor  120  reads the value of this “movement suspension” button to see if it is pressed or released. If the processor  120  detects that the button is pressed, then the processor  120  reports a motion value of zero to the video game console  140 . If the processor  120  detects that the button is released, then the processor  120  instead reports a motion value to the video game console  140  that is based on the current displacement value that is calculated by the image processor  125  (which is in turn proportional to the motion of the video camera  110 ).  
         [0027]     In this embodiment there is a button on the right hand side of the gun. This button is near the rear of the trigger guard. The button is placed such that it is located directly under the middle knuckle of the trigger finger of the game player&#39;s right hand. The game player can depress this button by pushing it with the underside of the knuckle of their trigger finger. This gives the game player a second button in addition to the trigger button on the gun that they can operate with their trigger finger. A third button that is operated by the game player&#39;s trigger finger is located on the interior of the trigger guard right above the actually trigger button. The game player can press this button simply by moving their trigger finger in an upward direction.  
         [0028]     In another embodiment of the present invention an expandable rifle stock is attached to the plastic enclosure. This allows the game player to brace the apparatus against their shoulder for greater precision and realism. Also, to enhance realism further, a solenoid is attached to the PCB, and it is instructed by the processor  120  to give a jolt every time the trigger button on the apparatus is pressed.  
         [0029]     In another embodiment of the present invention a pressure sensitive foot pad is attached to the PCB via a wired connection. The player stands on the foot pad and the processor  120  reads the steps that the user takes while on the pad. This processor  120 , for example, translates this foot step information into commands that cause the in-game character of a video game to walk in a particular direction. This increases the immersive-ness of the apparatus, and also can give the game player some exercise. A pulse rate sensor is also attached to the PCB, and the pulse rate sensor is connected to metal contacts on the pistol grip of the enclosure. The heart activity of the game player is monitored via this sensor via electrical signals from the user&#39;s skin that are conducted through the metal contacts. The user&#39;s pulse is reported on an LCD display (that is also attached to the PCB) and is viewable through a hole in the enclosure. This allows the game player to monitor the level of physical workout that they are receiving while playing the game and walking, running, and jumping on the foot pad.  
         [0030]     In another embodiment of the present invention, the rotation of the apparatus along an axis that runs through the front to the back of the camera  110 . The rotation information is used to keep track of which direction is up, and therefore keep the lateral movements of the apparatus consistent with a real gun. If this is not done then the direction of motion generated in the video game will be different depending upon the how the apparatus is tilted along its Z axis.  
         [0031]     In another embodiment of the present invention, a beacon is use in order to give the image processor  125  an easy to track reference point for tracking motion and rotation. Without the beacon the image processor  125  would have to rely on there being very trackable objects within every environment, and there is no guarantee of this. In this embodiment the beacon is in the form of a reflector. A light source is generated by an LED attached to the PCB. This light is reflected off of the reflector to produce a very trackable land mark for the image processor  125 .  
         [0032]     In another embodiment of the present invention an LED is contained in a separate enclosure from the gun shaped housing. The LED generates light that is detectable by the image processor  125 . The image of this light is then found in images from the camera  110 , and its displacement from image to image is calculated. This displacement is translated to a movement action within the video game and transferred to the video game via the video game controller plug  130 . The orientation of the LED is fixed, so the image processor  125  can also determine the rotation of the camera  110  about the axis that runs through the camera  110  by examining the degree of tilt of the LED beacon within the images.  
         [0033]     In another embodiment of the present invention the image processor  125  examines images from the camera  110  to look for the telltale signs of a television. Once the television is detected then is it tracked in subsequent images. The displacement of the television across images is calculated and translated to motion information that the video game console  140  can understand, and this motion information is sent to the video game console  140  to cause movement of an in game character. The rotation and orientation of the gun about its Z axis is also calculated based on the assumption that the top of the television is facing upward. The detection of the television is also used by the image processor  125  to disregard any motion that occurs with the television, since this motion will probably be contrary to the motion of the apparatus, and therefore would ruin the displacement calculations.  
         [0034]     In another embodiment of the present invention a second camera is attached to the PCB, and is aimed toward the game player. The images from this second camera are analyzed to identify the game player. The image of the game player is analyzed to detect specific movements and gestures of the game player. These gestures are then mapped to codes for functions of the video game. These codes are then transferred to the video game to invoke specific actions within the video game based on the motions of the game player&#39;s body. One of the gestures a user can perform is to kneel down. In many video games, especially FPS games, there is a button on a gamepad controller (or key on the keyboard of a PC) that is used to cause the in game character to crouch. When the game player kneels down, then the image processor  125  will identify this action, and the processor  120  will map this action to an appropriate code for the video game, thereby causing the in game character on the screen to also kneel down.  
         [0035]     In another embodiment of the present invention even greater precision of movement of the features being tracked can be determined by doing a “sub pixel” search. If an image is HEIGHT pixels high and WIDTH pixels wide, then the image contains resolution of HEIGHT×WIDTH pixels. But also, grayscale and color images contain additional resolution in their “color depth”. Color depth is determined by the number of bits that are used to represent each pixel. In a monochrome bitmap only 1 bit is used per pixel. But in this embodiment  24  bits are used to represent each pixel. If a literal image compare is done between the feature to search for, and the image frame, then this 24 bits of color resolution is not being fully utilized. This is because when a camera moves only a tiny amount then this can cause the features in the image to simply bleed a bit to the next pixel without moving a whole pixel in any direction. With a direct feature compare from pixels in the feature to pixels in the image this small motion will not be detected. In order to harness this color depth resolution, and therefore detect this small movement of the video camera  110 , the feature being searched for has to be shifted a fraction of a pixel in a given direction. This is done by looking at each pixel of the feature bitmap, and then giving it a new color value that is based on its neighboring pixels in the direction the pixel is being shifted. For example, if the pixels are being shifted 0.5 of a pixel directly to the right, then the new pixel value will be 50% of the current pixel color value plus 50% of the color value of the pixel directly to the right. If the pixels are being shifted in a diagonal direction then the new value of the pixel is based on the sum of the fractional values of pixels that the shifted pixel overlaps when it is shifted (including the contribution from the old location of the shifted pixel).  
         [0036]     Note that the invention described herein is not limited to the aforementioned embodiments, but rather these are examples meant to help crystallize the invention in the mind of the reader. For example, the device itself can be connected to the video game console  140  via many physical and wireless means, and the camera used could be of many different varieties of camera, including infrared, etc. And the video game could reside on any video game player, including many different video game console  140  devices and personal computers.