Display device, game system, and game process method

A game system includes a home-console type game device, a controller device, and a portable display device. The portable display device includes an infrared light-emitting portion capable of emitting infrared light. The controller device wirelessly transmits to the game device operation date including data representing a detection result by an image-capturing section, data representing a detection result by an inertia sensor, and data representing an operation performed on an operation button. The game device receives the operation data from the controller device, and performs a game process based on the operation data.

This application is the U.S. national phase of International Application No. PCT/JP2011/000566 filed 2 Feb. 2011 which designated the U.S. and claims priority to JP Patent Application No. 2010-022022 filed 3 Feb. 2010, JP Patent Application No. 2010-022023 filed 3 Feb. 2010, JP Patent Application No. 2010-177893 filed 6 Aug, 2010, JP Patent Application No. 2010-185315 filed 20 Aug. 2010, JP Patent Application No. 2010-192220 filed 30 Aug. 2010, JP Patent Application No. 2010-192221 filed 30 Aug. 2010, JP Patent Application No. 2010-245298 filed 1 Nov. 2010 and JP Patent Application No. 2010-245299 filed 1 Nov. 2010, the entire contents of each of which are hereby incorporated by reference.

TECHNICAL FIELD

The present invention relates to a display device for displaying a game image, and the like, which is a portable display device that can be carried around by the user, and to a game system and a game process method using such a display device.

BACKGROUND ART

There are conventional game systems in which the player can perform a game operation by moving the controller device itself (e.g., see Patent Document 1). For example, a game system described in Patent Document 1 uses the image-capturing means of the controller device to capture an image of a marker device, and uses information obtained from the image captured (captured image) as a game input, thus allowing for an operation of moving the controller device itself. Specifically, since it is possible to determine the position or the attitude of the controller device from the position of the marker device in the captured image, the position of the marker device is obtained as the information described above to perform a game process in accordance with the position of the marker device in the game system. Note that in the game system, while the position at which the marker device is placed is arbitrary, the marker device is normally placed around the display device because it feels natural for the user to use the controller device while pointing it toward the screen on which the game image is displayed.

CITATION LIST

Patent Document

SUMMARY OF INVENTION

Technical Problem

As described above, in the game system, the user uses the controller device while pointing it toward the display device. The game system typically uses a non-portable, fixedly-installed display device such as a television. Therefore, where the captured image is used as an input, the user needs to use the controller device while always pointing it toward the fixedly-installed display device. That is, with the game system, the orientation in which the controller device can be used may be limited, and there is room for improvement with the degree of freedom in operations to be performed on the controller device.

It is therefore an object of the present invention to provide a display device, a game system and a game process method with which it is possible to improve the degree of freedom in operations to be performed on the controller device.

Solution to Problem

The present invention employs the following configurations (1) to (14) to attain the object mentioned above.

An example of the present invention is a game system including a home-console type game device, a controller device, and a portable display device.

The game device includes an operation data reception section, a game process section, a game image generation section, a game image compression section, and a data transmission section. The operation data reception section receives operation data from the controller device. The game process section performs a game process based on the operation data. The game image generation section successively generates a first game image based on the game process. The game image compression section successively compresses the first game image to generate compressed image data. The data transmission section successively wirelessly transmits the compressed image data to the portable display device.

The controller device includes an image-capturing section, an inertia sensor, at least one operation button, and an operation data transmission section. The image-capturing section is capable of detecting infrared light. The operation data transmission section wirelessly transmits the operation data to the game device, wherein the operation data includes data representing a detection result from the image-capturing section, data representing a detection result from the inertia sensor, and data representing an operation performed on the operation button.

The portable display device includes an infrared light-emitting portion, a game image reception section, a game image expansion section, and a display section. The infrared light-emitting portion is capable of emitting infrared light. The game image reception section successively receives the compressed image data from the game device. The game image expansion section successively expands the compressed image data to obtain the first game image. The display section successively displays the first game image obtained by the expansion.

The “game device” may be any device as long as it is an information processing device capable of performing a game process, and generating an image based on the game process. The game device may be a single-purpose information processing device for games, or a general-purpose information processing device such as an ordinary personal computer.

The “controller device” may further include other components, in addition to the components above. For example, the controller device may further include display means and sound outputting means. The “detection result from the image-capturing section” may be the image itself captured by the image-capturing section, or information obtained from the image (e.g., the marker coordinates to be described later, etc.).

The term “portable” means it has such a size that it can be held in hand and moved around by the user, and the position thereof can be moved to an arbitrary position. Note however that the “portable display device” may be used while being moved in the game as in for example the first to fourth game examples to be described later, or may be used while being fixedly placed (not moved) in the game as in for example the fifth game example to be described later.

The “game system” only needs to include the game device, the controller device, and the portable display device, and may or may not include the external display device displaying the second game image to be described later. That is, the game system may be provided in a form where the external display device is not included or in a form where it is included.

With the configuration (1) above, the portable display device includes the infrared light-emitting portion and the display section, and the controller device includes the image-capturing section, wherein the game process is performed based on the detection result from the image-capturing section. Therefore, the user can perform a game operation by pointing the image-capturing section of the controller device toward the infrared light-emitting portion of the portable display device. Here, since the display device is portable, the user can place the display device at an arbitrary position, and can therefore use the controller device while pointing it in an arbitrary orientation. Thus, with the configuration (1) above, it is possible to improve the degree of freedom in operations to be performed on the controller device, as compared with conventional game systems where the orientation in which the controller device can be used may be limited.

With the configuration (1) above, the portable display device only needs to perform at least the expansion process for the image data, and the game process may be performed on the game device side. Even if the game process becomes more complicated, it only increases the computation on the game device side, and does not substantially influence the amount of computation of the image expansion process by the portable display device. Therefore, even if a complicated game process is required, the computational load on the portable display device side can be kept within a predetermined range, and the portable display device is not required of high information processing capabilities. This makes it easier to reduce the size and weight of the portable display device, and makes it easier to manufacture the portable display device.

Moreover, with the configuration (1) above, since the first game image is transmitted, in a compressed form, from the game device to the portable display device, the game image can be wirelessly transmitted at a high speed, and the delay from when the game process is performed until the game image is displayed is kept small.

The game system may further include a marker device capable of emitting infrared light. Then, the game image generation section further generates a second game image based on the game process. The game device further includes an image outputting section and a light emission control section. The image outputting section successively outputs the second game image to an external display device which is separate from the portable display device. The light emission control section controls the light emission of the marker device.

The “external display device” only needs to be separate from the portable display device, and may include any device, in addition to a television2in the embodiment to be described later, as long as it is capable of displaying the second game image generated by the game device. For example, the external display device may be formed as an integral unit (in a single casing) with the game device.

With the configuration (2) above, the game system includes a marker device separate from the infrared light-emitting portion of the portable display device, and the second game image is displayed on an external display device separate from the portable display device on which the first game image is displayed. Thus, where the marker device is placed around the external display device, the user can use the controller device while pointing it to either one of the two display devices. That is, since the user can operate the controller device while pointing it to either one of the two display devices, the degree of freedom in operations to be performed on the controller device is further improved.

Moreover, with the configuration (2) above, since the portable display device can be placed arbitrarily, it is possible to arbitrarily set the positional relationship between the two display devices. Therefore, by placing the two display devices at appropriate positions in accordance with the content of the game, it is possible to realize a more realistic operation using the controller device, and to realize a game with better playability (see the fifth game example to be described later). By changing the positional relationship between the two display devices as necessary, it is possible to accommodate various games where two display devices are used in various positional relationships.

With the configuration (2) above, since the second game image can be displayed on the external display device, two different types of game images can be presented to the player. Therefore, the game space can be expressed in various methods with two types of game images. Thus, with the configuration (2) above, it is possible to present to the player game images that are easier to view and easier to perform game operations with.

The game system may include two controller devices. Then, the game process section performs the game process based on the operation data received from the controller devices.

With the configuration (3) above, a game operation can be performed while pointing one controller device toward the portable display device, and a game operation can be performed while pointing the other controller device toward the marker device. Therefore, where the marker device is placed around the external display device, two players can simultaneously play a game (e.g., the fifth game example to be described later) in which the controller device and the display device are used as a set.

The light emission control section may control the light emission of the marker device and the infrared light-emitting portion in accordance with content of the game process.

To “control (the light emission of the marker device and the infrared light-emitting portion) in accordance with content of the game process” includes to control the light emission in accordance with the type of the game program executed by the game device, and to control the light emission in accordance with the game status (the object being controlled by the player, the manner in which the object is being controlled, or the status of the game's progress) during the execution of the same game program.

With the configuration (4) above, the game device can control which one of the marker device and the infrared light-emitting portion is to be lit (or both are to be lit) in accordance with the content of the game process. Here, depending on the content of the game, only one of the two light-emitting devices, i.e., the marker device and the infrared light-emitting portion of the portable display device, may be used. Where two light-emitting devices are both lit, one may not be able to accurately perform an operation using the controller device because the game device cannot determine from which light-emitting device the image-capturing section of the controller device is detecting the infrared light. In contrast, with the configuration (4) above, light can be emitted from an appropriate one of the two light-emitting devices in accordance with the content of the game process, and it is therefore possible to accommodate various games, and to prevent an operation with the controller device from being inaccurate due to erroneous detection.

The light emission control section may generate control data representing control instructions for the light emission of the infrared light-emitting portion. Then, the data transmission section wirelessly transmits the control data to the portable display device. The portable display device further includes a control data reception section for receiving the control data from the game device. The infrared light-emitting portion operates based on the received control data.

The “data transmission section” may transmit the “control data” together with the image data, or may transmit it at a different point in time from the transmission of the image data. That is, even where the image data are successively transmitted, transmit data are transmitted only when necessary, and do not need to be successively transmitted together with the image data.

With the configuration (5) above, the game device can easily control the light emission of the infrared light-emitting portion by transmitting the control data to the portable display device.

An example of the present invention is a portable display device capable of wirelessly communicating with a game device. The game device receives, from a controller device including an image-capturing section capable of detecting infrared light, data representing a detection result of the image-capturing section, and successively transmits, to the display device, compressed image data which is obtained by compressing a game image generated based on a game process performed based on the data.

The display device includes an infrared light-emitting portion, a game image reception section, a game image expansion section, and a display section. The infrared light-emitting portion is capable of emitting infrared light. The game image reception section successively receives the compressed image data from the game device. The game image expansion section successively expands the compressed image data to obtain the game image. The display section successively displays the game image obtained by the expansion.

With the configuration (6) above, as with the configuration (1) above, the portable display device includes the infrared light-emitting portion and the display section, and the controller device includes the image-capturing section, wherein the game process is performed based on the detection result from the image-capturing section. Therefore, as with the configuration (1) above, the user can place the display device at an arbitrary position, and can therefore use the controller device while pointing it in an arbitrary orientation, thus improving the degree of freedom in operations to be performed on the controller device.

With the configuration (6) above, as with the configuration (1) above, the portable display device is not required of high information processing capabilities, thus making it easier to reduce the size and weight of the portable display device, and making it easier to manufacture the portable display device. With the configuration (6) above, since the game image is transmitted, in a compressed form, from the game device to the portable display device, the game image can be wirelessly transmitted at a high speed, and the delay from when the game process is performed until the game image is displayed is kept small.

The display device may further include a touch panel, an inertia sensor, and an operation data transmission section. The touch panel is provided on a screen of the display section. The operation data transmission section wirelessly transmits to the game device operation data including output data of the touch panel and the inertia sensor. Then, the game device performs the game process based on the operation data.

With the configuration (7) above, the portable display device can function also as a controller device. For example, where the display device is used in the game system, the user can perform an operation by moving the display device itself while looking at the screen of the display section, or the display device can be used as display means such that it is placed at an arbitrary position and another controller device is used while being pointed toward the display device. That is, with the configuration (7) above, a multi-purpose device can be provided as it can be used either as a controller device or as a display device.

The game device may wirelessly transmit a game sound generated based on the game process to the display device. Then, the game device further includes a game sound reception section for receiving the game sound from the game device; and a speaker for outputting the game sound received by the game sound reception section.

With the configuration (8) above, the game sound wirelessly transmitted from the game device to the display device may be transmitted in a compressed form as in the embodiment to be described later, or may be transmitted in an uncompressed form.

With the configuration (8) above, as with the game image, the game sound can be outputted from the display device.

The portable display device may further include a microphone. Then, the operation data transmission section further wirelessly transmits data of sound detected by the microphone to the game device.

With the configuration (9) above, the data of sound wirelessly transmitted from the controller device to the game device may be transmitted in a compressed form as in the embodiment to be described later, or may be transmitted in an uncompressed form.

With the configuration (9) above, the sound (microphone sound) detected by the microphone of the controller device is transmitted to the game device. Therefore, the game device can use the microphone sound as the game sound, or use a result performing a sound recognition process on the microphone sound as the game input.

The display device may further include a camera and a camera image compression section. The camera image compression section compresses a camera image captured by the camera to generate compressed captured image data. Then, the operation data transmission section further wirelessly transmits the compressed captured image data to the game device.

With the configuration (10) above, the camera image captured by the camera of the display device is transmitted to the game device. Therefore, the game device may use the camera image as the game image, or use a result of performing an image recognition process on the camera image as the game input. With the configuration (5) above, since the camera image is transmitted in a compressed form, the camera image can be wirelessly transmitted at high speed.

The display device may include a plurality of front surface operation buttons, and direction input sections capable of specifying directions. The plurality of front surface operation buttons are provided on a front surface of the display device, on which a screen of the display section is provided, on opposite sides of the screen. The direction input sections are provided on the front surface on opposite sides of the screen. Then, the operation data further includes data representing operations performed on the plurality of front surface operation buttons and the direction input sections.

With the configuration (11) above, the operation buttons and the direction input sections are provided on opposite sides of the screen of the display device. Therefore, since the player can operate the operation buttons and the direction input sections while holding the display device (typically with the thumbs), it is possible to easily operate the operation buttons and the direction input sections even while performing the operation of moving the display device.

The display device may further include a plurality of back surface operation buttons and a plurality of side surface operation buttons. The plurality of back surface operation buttons are provided on a back surface of the display device. The back surface is a surface opposite to the front surface of the display device, on which the screen of the display section and the touch panel are provided. The plurality of side surface operation buttons are provided on a side surface between the front surface and the back surface. Then, the operation data further includes data representing operations performed on the plurality of back surface operation buttons and the side surface operation buttons.

With the configuration (12) above, operation buttons are provided on the back surface and the side surface of the display device. Therefore, since the player can operate these operation buttons while holding the display device (typically with the index fingers or the middle fingers), it is possible to easily operate the operation buttons even while performing the operation of moving the display device.

The display device may further include a magnetic sensor. Then, the operation data further includes data of a detection result of the magnetic sensor.

With the configuration (13) above, the display device includes the magnetic sensor, and the output result of the magnetic sensor is used in the game process in the game device. Therefore, the player can perform the game operation by moving the display device. Since the game device can determine the absolute attitude of the display device in the real space from the output result from the magnetic sensor, it is possible to accurately calculate the attitude of the display device by using the output result from the inertia sensor and the output result from the magnetic sensor, for example.

While the inertia sensor may be any inertia sensor, it may include, for example, a 3-axis acceleration sensor and a 3-axis gyrosensor.

With the configuration (14) above, by using two types of sensors, i.e., an acceleration sensor and a gyrosensor, as the inertia sensor, it is possible to accurately calculate the movement and the attitude of the portable display device.

Another example of the present invention may be embodied as a game process method carried out in the game system of the configurations (1) to (5).

ADVANTAGEOUS EFFECTS OF INVENTION

According to the present invention, a portable display device is provided with an infrared, light-emitting portion, and a game image is displayed on the display device so that it is possible to use a controller device while pointing it in an arbitrary direction, and it is possible to improve the degree of freedom in operations to be performed on the controller device.

DESCRIPTION OF EMBODIMENTS

[1. General Configuration of Game System]

A game system1according to an embodiment of the present invention will now be described with reference to the drawings.FIG. 1is an external view of the game system1. InFIG. 1, the game system1includes a non-portable display device (hereinafter referred to as a “television”)2such as a television receiver, a home-console type game device3, an optical disc4, a controller5, a marker device6, and a terminal device7. In the game system1, the game device3performs a game process based on a game operation performed by using the controller5, and a game image obtained through the game process is displayed on the television2and/or the terminal device7.

In the game device3, the optical disc4typifying an information storage medium used for the game device3in a replaceable manner is detachably inserted. An information processing program (typically, a game program) to be executed by the game device3is stored in the optical disc4. The game device3has, on the front surface thereof, an insertion opening for the optical disc4. The game device3reads and executes the information processing program stored in the optical disc4which is inserted through the insertion opening, so as to perform the game process.

The television2is connected to the game device3through a connecting cord. A game image obtained as a result of the game process performed by the game device3is displayed on the television2. The television2includes a speaker2a(FIG. 2), and the speaker2aoutputs a game sound obtained as a result of the game process. Note that in other embodiments, the game device3and the non-portable display device may be an integral unit. The communication between the game device3and the television2may be wireless communication.

The marker device6is provided along the periphery of the screen (on the upper side of the screen inFIG. 1) of the television2. The user (player) can perform a game operation of moving the controller5, the details of which will be described later, and the marker device6is used by the game device3for calculating the movement, the position, the orientation, etc., of the controller5. The marker device6includes two markers6R and6L on both ends thereof. Specifically, a marker6R (as well as the marker6L) includes one or more infrared LEDs (Light20. Emitting Diodes), and emits an infrared light forward from the television2. The marker device6is connected to the game device3, and the game device3is able to control the lighting of each infrared LED of the marker device6. Note that the marker device6is portable, and the user can install the marker device6at any position. WhileFIG. 1shows an embodiment where the marker device6is installed on top of the television2, the position and the direction of installment of the marker device6are not limited to this.

The controller5provides the game device3with operation data representing the content of an operation performed on the controller itself. The controller5and the game device3can communicate with each other by wireless communication. In the present embodiment, the wireless communication between the controller5and the game device3uses, for example, Bluetooth (Registered Trademark) technology. In other embodiments, the controller5and the game device3may be connected by a wired connection. While the number of the controllers5included in the game system1is one in the present embodiment, the game device3can communicate with a plurality of controllers, and a game can be played by multiple players by using a predetermined number of controllers at the same time. The detailed configuration of the controller5will be described later.

The terminal device7has such a size that it can be held in hand by the user, and the user can hold and move the terminal device7, or can use the terminal device7placed at an arbitrary position. The terminal device7, whose detailed configuration will be described later, includes an LCD (Liquid Crystal Display)51as display means, and input means (a touch panel52, a gyrosensor64, etc., to be described later). The terminal device7and the game device3can communicate with each other by wireless connection (or wired connection). The terminal device7receives data of an image (e.g., a game image) generated by the game device3from the game device3, and displays the image on the LCD51. Note that while an LCD is used as the display device in the present embodiment, the terminal device7may include any other display device such as a display device utilizing EL (Electro Luminescence), for example. The terminal device7transmits operation data representing the content of an operation performed on the terminal device itself to the game device3.

[2. Internal Configuration of Game Device3]

Next, an internal configuration of the game device3will be described with reference toFIG. 2.FIG. 2is a block diagram illustrating an internal configuration of the game device3. The game device3includes a CPU (Central Processing Unit)10, a system LSI11, an external main memory12, a ROM/RTC13, a disc drive14, an AV-IC15, and the like.

The CPU10performs game processes by executing a game program stored in the optical disc4, and functions as a game processor. The CPU10is connected to the system LSI11. To the system LSI11, the external main memory12, the ROM/RTC13, the disc drive14, and the AV-IC15as well as the CPU10are connected. The system LSI11performs processes for controlling data transmission between the respective components connected thereto, generating an image to be displayed, acquiring data from an external device, and the like. The internal configuration of the system LSI11will be described below. The external main memory12of a volatile type stores a program such as a game program read from the optical disc4and a game program read from a flash memory17, and various data, and the external main memory12is used as a work area and a buffer area for the CPU10. The ROM/RTC13includes a ROM (a so-called boot ROM) incorporating a boot program for the game device3, and a clock circuit (RTC: Real Time Clock) for counting time. The disc drive14reads program data, texture data, and the like from the optical disc4, and writes the read data into an internal main memory11eto be described below or the external main memory12.

The system LSI11includes an input/output processor (I/O processor)11a, a GPU (Graphics Processor Unit)11b, a DSP (Digital Signal Processor)11c, a VRAM (Video RAM)11d, and the internal main memory11e. Although not shown in the figures, these components11ato11eare connected with each other through an internal bus.

The GPU11b, acting as a part of rendering means, generates an image in accordance with a graphics command (rendering command) from the CPU10. The VRAM11dstores data (data such as polygon data and texture data) necessary for the GPU11bto execute the graphics command. When an image is generated, the GPU11bgenerates image data using data stored in the VRAM11d. Note that in the present embodiment, the game device3generates both the game image displayed on the television2and the game image displayed on the terminal device7. Hereinafter, the game image displayed on the television2may be referred to as the “television game image”, and the game image displayed on the terminal device7may be referred to as the “terminal game image”.

The DSP11c, functioning as an audio processor, generates sound data using sound data and sound waveform (tone quality) data stored in the internal main memory11eor the external main memory12. Note that in the present embodiment, also with the game sound as with the game image, there are generated both a game sound outputted from the speaker of the television2and a game sound outputted from the speaker of the terminal device7. Hereinafter, the game sound outputted from the television2may be referred to as a “television game sound”, and the game sound outputted from the terminal device7may be referred to as a “terminal game sound”.

As described above, of the images and sounds generated in the game device3, data of the image and sound outputted from the television2is read out by the AV-IC15. The AV-IC15outputs the read image data to the television2via an AV connector16, and outputs the read sound data to the speaker2aprovided in the television2. Thus, an image is displayed on the television2, and a sound is outputted from the speaker2a.

Of the images and sounds generated in the game device3, data of the image and sound outputted from the terminal device7is transmitted to the terminal device7by an input/output processor11e, etc. The data transmission to the terminal device7by the input/output processor11a, or the like, will be described later.

The input/output processor11aexchanges data with components connected thereto, and downloads data from an external device. The input/output processor11ais connected to the flash memory17, a network communication module18, a controller communication module19, an extension connector20, a memory card connector21, and a codec LSI27. An antenna22is connected to the network communication module18. An antenna23is connected to the controller communication module19. The codec LSI27is connected to a terminal communication module28, and an antenna29is connected to the terminal communication module28.

The game device3can be connected to a network such as the Internet to communicate with an external information processing device (e.g., other game devices, various servers, etc.). That is, the input/output processor11acan be connected to a network such as the Internet via the network communication module18and the antenna22to communicate with an external information processing device connected to the network. The input/output processor11aregularly accesses the flash memory17, and detects the presence or absence of any data which needs to be transmitted to the network, and when detected, transmits the data to the network via the network communication module18and the antenna22. Further, the input/output processor11ereceives data transmitted from an external information processing device and data downloaded from a download server via the network, the antenna22and the network communication module18, and stores the received data in the flash memory17. The CPU10executes a game program so as to read data stored in the flash memory17and use the data on the game program. The flash memory17may store save data (game result data or unfinished game data) of a game played using the game device3in addition to data exchanged between the game device3and an external information processing device. The flash memory17may store a game program.

The game device3can receive operation data from the controller5. That is, the input/output processor11areceives operation data transmitted from the controller5via the antenna23and the controller communication module19, and stores (temporarily stores) it in a buffer area of the internal main memory11eor the external main memory12.

The game device3can exchange data such as images and sounds with the terminal device7. When transmitting a game image (terminal game image) to the terminal device7, the input/output processor11aoutputs data of a game image generated by the GPO11bto the codec LSI27. The codec LSI27performs a predetermined compression process on the image data from the input/output processor11a. The terminal communication module28wirelessly communicates with the terminal device7. Therefore, image data compressed by the codec LSI27is transmitted by the terminal communication module28to the terminal device7via the antenna29. Note that in the present embodiment, the image data transmitted from the game device3to the terminal device7is image data used in a game, and the playability of a game is adversely influenced if there is a delay in the image displayed in the game. Therefore, it is preferred to eliminate delay as much as possible for the transmission of image data from the game device3to the terminal device7. Therefore, in the present embodiment, the codec LSI27compresses image data by using a compression technique with high efficiency such as the H.264 standard, for example. Note that other compression techniques may be used, and image data may be transmitted uncompressed if the communication speed is sufficient. The terminal communication module28is, for example, a Wi-Fi certified communication module, and may perform wireless communication at high speed with the terminal device7using a MIMO (Multiple Input Multiple Output) technique employed in the IEEE802.11n standard, for example, or may use other communication schemes.

The game device3transmits sound data to the terminal device7, in addition to image data. That is, the input/output processor11aoutputs sound data generated by the DSP11cto the terminal communication module28via the codec LSI27. The codec LSI27performs a compression process on sound data, as with image data. While the compression scheme for sound data may be any scheme, it is preferably a scheme with high compression ratio and little sound deterioration. In other embodiments, the sound data may be transmitted uncompressed. The terminal communication module28transmits the compressed image data and sound data to the terminal device7via the antenna29.

Moreover, the game device3transmits various control data to the terminal device7as necessary, in addition to the image data and the sound data. Control data is data representing control instructions for components of the terminal device7, and it for example represents an instruction for controlling the lighting of a marker section (a marker section55shown inFIG. 10), an instruction for controlling the image-capturing operation of a camera (a camera56shown inFIG. 10), etc. The input/output processor11atransmits control data to the terminal device7in response to an instruction of the CPU10. Note that while the codec LSI27does not perform a data compression process in the present embodiment for the control data, it may perform a compression process in other embodiments. Note that the above-described data transmitted from the game device3to the terminal device7may be encrypted as necessary or may not be encrypted.

The game device3can receive various data from the terminal device7. In the present embodiment, the terminal device7transmits operation data, image data and sound data, the details of which will be described later. Data transmitted from the terminal device7are received by the terminal communication module28via the antenna29. The image data and the sound data from the terminal device7are subjected to a compression process similar to that on the image data and the sound data from the game device3to the terminal device7. Therefore, these image data and sound data are sent from the terminal communication module28to the codec LSI27, and subjected to an expansion process by the codec LSI27to be outputted to the input/output processor11a. On the other hand, the operation data from the terminal device7may not be subjected to a compression process since the amount of data is small as compared with images and sounds. It may be encrypted as necessary, or it may not be encrypted. Therefore, after being received by the terminal communication module28, the operation data is outputted to the input/output processor11avia the codec LSI27. The input/output processor11astores (temporarily stores) data received from the terminal device7in a buffer area of the internal main memory11eor the external main memory12.

The game device3can be connected to another device or an external storage medium. That is, the input/output processor11ais connected to the extension connector20and the memory card connector21. The extension connector20is a connector for an interface, such as USB or SCSI. The extension connector20can receive a medium such as an external storage medium, a peripheral device such as another controller, or a wired communication connector which enables communication with a network in place of the network communication module18. The memory card connector21is a connector for connecting thereto an external storage medium such as a memory card. For example, the input/output processor11acan access an external storage medium via the extension connector20or the memory card connector21to store data in the external storage medium or read data from the external storage medium.

The game device3includes a power button24, a reset button25, and an eject button26. The power button24and the reset button25are connected to the system LSI11. When the power button24is on, power is supplied to the components of the game device3from an external power supply through an AC adaptor not shown. When the reset button25is pressed, the system LSI11reboots a boot program of the game device3. The eject button26is connected to the disc drive14. When the eject button26is pressed, the optical disc4is ejected from the disc drive14.

Note that in other embodiments, some of the components of the game device3may be provided as extension devices separate from the game device3. Then, an extension device may be connected to the game device3via the extension connector20, for example. Specifically, an extension device may include components of the codec LSI27, the terminal communication module28and the antenna29, for example, and can be attached/detached to/from the extension connector20. Thus, by connecting the extension device to a game device which does not include the above components, the game device can communicate with the terminal device7.

[3. Configuration of Controller5]

Next, with reference toFIGS. 3 to 7, the controller5will be described.FIG. 3is a perspective view illustrating an external configuration of the controller5.FIG. 4is a perspective view illustrating an external configuration of the controller5.FIG. 3is a perspective view showing the controller5as viewed from the top rear side thereof, andFIG. 4is a perspective view showing the controller5as viewed from the bottom front side thereof.

As shown inFIGS. 3 and 4, the controller5has a housing31formed by, for example, plastic molding. The housing31has a generally parallelepiped shape extending in a longitudinal direction from front to rear (Z-axis direction shown inFIG. 3), and as a whole is sized to be held by one hand of an adult or a child. A user can perform game operations by pressing buttons provided on the controller5, and moving the controller5itself to change the position and the orientation (tilt) thereof.

The housing31has a plurality of operation buttons. As shown inFIG. 3, on the top surface of the housing31, a cross button32a, a first button32b, a second button32c, an A button32d, a minus button32e, a home button32f, a plus button32g, and a power button32hare provided. In the present specification, the top surface of the housing31on which the buttons32ato32hare provided may be referred to as a “button surface”. On the other hand, as shown inFIG. 4, a recessed portion is formed on the bottom surface of the housing31, and a B button32iis provided on a rear slope surface of the recessed portion. The operation buttons32ato32iare assigned, as necessary, their respective functions in accordance with the game program executed by the game device3. Further, the power button32his intended to remotely turn ON/OFF the power of the main unit of the game device3. The home button32fand the power button32heach have the top surface thereof recessed below the top surface of the housing31. Therefore, the home button32fand the power button32hare prevented from being inadvertently pressed by the user.

On the rear surface of the housing31, the connector33is provided. The connector33is used for connecting another device (e.g., another sensor unit or another controller) to the controller5. Both sides of the connector33on the rear surface of the housing31have a fastening hole33afor preventing easy disengagement of another device as described above.

In the rear-side portion of the top surface of the housing31, a plurality (four inFIG. 3) of LEDs34ato34dare provided. The controller5is assigned a controller type (number) so as to be distinguishable from other controllers. The LEDs34ato34dare each used for informing the user of the controller type which is currently being set for the controller5, and for informing the user of the battery level of the controller5, for example. Specifically, when a game operation is performed using the controller5, one of the plurality of LEDs34ato34dcorresponding to the controller type is lit up.

The controller5has an image capturing/processing section35(FIG. 6), and alight incident surface35aof the image capturing/processing section35is provided on the front surface of the housing31, as shown inFIG. 4. The light incident surface35ais made of a material transmitting therethrough at least infrared light from the markers6R and6L.

On the top surface of the housing31, sound holes31afor externally outputting a sound from a speaker47(FIG. 5) provided in the controller5are provided between the first button32band the home button32f.

Next, with reference toFIGS. 5 and 6, an internal structure of the controller5will be described.FIGS. 5 and 6are diagrams illustrating the internal structure of the controller5. Note thatFIG. 5is a perspective view illustrating a state where an upper casing (a part of the housing31) of the controller5is removed.FIG. 6is a perspective view illustrating a state where a lower casing (a part of the housing31) of the controller5is removed. The perspective view ofFIG. 6shows a substrate30ofFIG. 5as viewed from the reverse side.

As shown inFIG. 5, the substrate30is fixed inside the housing31, and on a top main surface of the substrate30, the operation buttons32ato32h, the LEDs34ato34d, an acceleration sensor37, an antenna45, the speaker47, and the like are provided. These elements are connected to a microcomputer42(seeFIG. 6) via lines (not shown) formed on the substrate30and the like. In the present embodiment, the acceleration sensor37is provided on a position offset from the center of the controller5with respect to the X-axis direction. Thus, calculation of the movement of the controller5being rotated about the Z-axis is facilitated. Further, the acceleration sensor37is provided anterior to the center of the controller5with respect to the longitudinal direction (Z-axis direction). Further, a wireless module44(FIG. 6) and the antenna45allow the controller5to act as a wireless controller.

On the other hand, inFIG. 6, at a front edge of a bottom main surface of the substrate30, the image capturing/processing section35is provided. The image capturing/processing section35includes an infrared filter38, a lens39, an image capturing element40and an image processing circuit41located in this order from the front of the controller5. These components38to41are attached on the bottom main surface of the substrate30. On the bottom main surface of the substrate30, the microcomputer42and a vibrator46are provided. The vibrator46is, for example, a vibration motor or a solenoid, and is connected to the microcomputer42via lines formed on the substrate30or the like. The controller5is vibrated by actuation of the vibrator46based on a command from the microcomputer42. Therefore, the vibration is conveyed to the user's hand holding the controller5, and thus a so-called vibration-feedback game is realized. In the present embodiment, the vibrator46is disposed slightly toward the front of the housing31. That is, the vibrator46is positioned offset from the center toward the end of the controller5so that the vibration of the vibrator46greatly vibrates the entire controller5. Further, the connector33is provided at the rear edge of the bottom main surface of the substrate30. In addition to the components shown inFIGS. 5 and 6, the controller5includes a quartz oscillator for generating a reference clock of the microcomputer42, an amplifier for outputting a sound signal to the speaker47, and the like.

The shape of the controller5, the shape of each operation button, the number and the positions of acceleration sensors and vibrators, and so on, shown inFIGS. 3 to 6are merely illustrative, and the present invention can be realized with other shapes, numbers, and positions. Further, although in the present embodiment the image-capturing direction of the image-capturing means is the Z-axis positive direction, the image-capturing direction may be any direction. That is, the position of the image capturing/processing section35(the light incident surface35aof the image capturing/processing section35) in the controller5may not be on the front surface of the housing31, but may be on any other surface on which light can be received from the outside of the housing31.

FIG. 7is a block diagram illustrating a configuration of the controller5. The controller5includes an operation section32(the operation buttons32ato32i), the image capturing/processing section35, a communication section36, the acceleration sensor37, and a gyrosensor48. The controller5transmits, as operation data, data representing the content of an operation performed on the controller itself, to the game device3. Note that hereinafter, the operation data transmitted by the controller5may be referred to as the “controller operation data”, and the operation data transmitted by the terminal device7may be referred to as the “terminal operation data”.

The operation section32includes the operation buttons32ato32idescribed above, and outputs, to the microcomputer42of the communication section36, operation button data indicating the input status of the operation buttons32ato32i(whether or not the operation buttons32ato32iare pressed).

The image capturing/processing section35is a system for analyzing image data captured by the image-capturing means to determine an area having a high brightness therein and calculate the centroid, the size, etc., of the area. The image capturing/processing section35has a maximum sampling period of, for example, about 200 frames/sec., and therefore can trace and analyze even a relatively fast motion of the controller5.

The image capturing/processing section35includes the infrared filter38, the lens39, the image capturing element40and the image processing circuit41. The infrared filter38transmits therethrough only infrared light included in the light incident on the front surface of the controller5. The lens39collects the infrared light transmitted through the infrared filter38so that it is incident on the image capturing element40. The image capturing element40is a solid-state image-capturing device such as, for example, a CMOS sensor or a CCD sensor, which receives the infrared light collected by the lens39, and outputs an image signal. The marker section55of the terminal device7and the marker device6of which images are captured are formed by markers outputting infrared light. Therefore, the provision of the infrared filter38enables the image capturing element40to receive only the infrared light transmitted through the infrared filter38and generate image data, so that an image of the image-capturing object (the marker section55and/or the marker device6) can be captured more accurately. Hereinafter, the image taken by the image capturing element40is referred to as a captured image. The image data generated by the image capturing element40is processed by the image processing circuit41. The image processing circuit41calculates the positions of the image-capturing objects within the captured image. The image processing circuit41outputs coordinates of the calculated positions, to the microcomputer42of the communication section36. The data representing the coordinates is transmitted as operation data to the game device3by the microcomputer42. Hereinafter, the coordinates are referred to as “marker coordinates”. The marker coordinates change depending on the orientation (tilt angle) and/or the position of the controller5itself, and therefore the game device3can calculate the orientation and the position of the controller5using the marker coordinates.

Note that in other embodiments, the controller5may not include the image processing circuit41, and the captured image itself may be transmitted from the controller5to the game device3. At this time, the game device3may have a circuit or a program, having the same function as the image processing circuit41, for calculating the marker coordinates.

The acceleration sensor37detects accelerations (including gravitational acceleration) of the controller5, that is, force (including gravity) applied to the controller5. The acceleration sensor37detects a value of acceleration (linear acceleration) in the straight line direction along the sensing axis direction, among all the acceleration applied to the detection section of the acceleration sensor37. For example, a multi-axis acceleration sensor having two or more axes detects acceleration components along the axes, as the acceleration applied to the detection section of the acceleration sensor. Note that while the acceleration sensor37is assumed to be an electrostatic capacitance type MEMS (Micro Electro Mechanical System) acceleration sensor, it may be another type of an acceleration sensor.

In the present embodiment, the acceleration sensor37detects linear acceleration in each of three axis directions, i.e., the up/down direction (Y-axis direction shown inFIG. 3), the left/right direction (the X-axis direction shown inFIG. 3), and the forward/backward direction (the Z-axis direction shown inFIG. 3), relative to the controller5. The acceleration sensor37detects acceleration in the straight line direction along each axis, and an output from the acceleration sensor37therefore represents a value of the linear acceleration for each of the three axes. In other words, the detected acceleration is represented as a three-dimensional vector in an XYZ-coordinate system (controller coordinate system) defined relative to the controller5.

Data (acceleration data) representing the acceleration detected by the acceleration sensor37is outputted to the communication section36. Note that the acceleration detected by the acceleration sensor37changes depending on the orientation (tilt angle) and the movement of the controller5itself, and therefore the game device3is allowed to calculate the orientation and the movement of the controller5using the obtained acceleration data. In the present embodiment, the game device3calculates the attitude, the tilt angle, etc., of the controller5based on the obtained acceleration data.

One skilled in the art will readily understand from the description herein that additional information relating to the controller5can be estimated or calculated (determined) through a process by a computer, such as a processor (for example, the CPU10) of the game device3or a processor (for example, the microcomputer42) of the controller5, based on an acceleration signal outputted from the acceleration sensor37(this applies also to an acceleration sensor63to be described later). For example, in the case where the computer performs a process on the premise that the controller5including the acceleration sensor37is in static state (that is, in the case where the process is performed on the premise that the acceleration to be detected by the acceleration sensor includes only the gravitational acceleration), when the controller5is actually in static state, it is possible to determine whether or not, or how much the attitude of the controller5is tilting relative to the direction of gravity, based on the detected acceleration. Specifically, when the state where the detection axis of the acceleration sensor37faces vertically downward is used as a reference, whether or not the controller3is tilting relative to the reference can be determined based on whether or not15(gravitational acceleration) is present, and the degree of tilt of the controller5relative to the reference can be determined based on the magnitude thereof. Further, with the multi-axis acceleration sensor37, it is possible to more specifically determine the degree of tilt of the controller5relative to the direction of gravity by performing a process on the acceleration signals of different axes. In this case, the processor may calculate, based on the output from the acceleration sensor37, the tilt angle of the controller5, or the tilt direction of the controller5without calculating the tilt angle. Thus, by using the acceleration sensor37in combination with the processor, it is possible to determine the tilt angle or the attitude of the controller5.

On the other hand, when it is premised that the controller is in dynamic state (where the controller5is being moved), the acceleration sensor37detects the acceleration based on the movement of the controller5, in addition to the gravitational acceleration, and it is therefore possible to determine the movement direction of the controller5by removing the gravitational acceleration component from the detected acceleration through a predetermined process. Even when it is premised that the controller5is in dynamic state, it is possible to determine the tilt of the controller5relative to the direction of gravity by removing the acceleration component based on the movement of the acceleration sensor from the detected acceleration through a predetermined process. Note that in other embodiments, the acceleration sensor37may include an embedded processor or another type of dedicated processor for per forming a predetermined process on an acceleration signal detected by the built-in acceleration detection means before the acceleration signal is outputted to the microcomputer42. For example, when the acceleration sensor37is used to detect static acceleration (for example, gravitational acceleration), the embedded or dedicated processor may convert the acceleration signal to a tilt angle (or another preferable parameter).

The gyrosensor48detects angular velocities about three axes (the X, Y and Z axes in the present embodiment). In the present specification, with respect to the image-capturing direction the Z-axis positive direction) of the controller5, the rotation direction about the X axis is referred to as the pitch direction, the rotation direction about the Y axis as the yaw direction, and the rotation direction about the Z axis as the roll direction. The number and combination of gyrosensors to be used are not limited to any particular number and combination as long as the gyrosensor48can detect angular velocities about three axes. For example, the gyrosensor48may be a 3-axis gyrosensor, or angular velocities about three axes may be detected by combining together a 2-axis gyrosensor and a 1-axis gyrosensor. Data representing the angular velocity detected by the gyrosensor48is outputted to the communication section36. The gyrosensor48may be a gyrosensor that detects an angular velocity or velocities about one axis or two axes.

The communication section36includes the microcomputer42, a memory43, the wireless module44and the antenna45. The microcomputer42controls the wireless module44for wirelessly transmitting, to the game device3, data acquired by the microcomputer42while using the memory43as a storage area in the process.

Data outputted from the operation section32, the image capturing/processing section35, the acceleration sensor37, and the gyrosensor48to the microcomputer42are temporarily stored in the memory43. The data are transmitted as the operation data (controller operation data) to the game device3. At the time of the transmission to the controller communication module19of the game device3, the microcomputer42outputs the operation data stored in the memory43to the wireless module44. The wireless module44uses, for example, the Bluetooth (registered trademark) technology to modulate the operation data onto a carrier wave of a predetermined frequency, and radiates the low power radio wave signal from the antenna45. That is, the operation data is modulated onto the low power radio wave signal by the wireless module44and transmitted from the controller5. The controller communication module19of the game device3receives the low power radio wave signal. The game device3demodulates or decodes the received low power radio wave signal to obtain the operation data. Based on the operation data obtained from the controller5, the CPU10of the game device3performs the game process. Note that while the wireless transmission from the communication section36to the controller communication module19is sequentially performed with a predetermined cycle, since the game process is generally performed with a cycle of 1/60 sec (as one frame period), the transmission is preferably performed with a cycle less than or equal to this period. The communication section36of the controller5outputs, to the controller communication module19of the game device3, the operation data at a rate of once per 1/200 sec, for example.

As described above, as operation data representing an operation performed on the controller itself, the controller5can transmit marker coordinate data, acceleration data, angular velocity data, and operation button data. The game device3performs the game process using the operation data as a game input. Therefore, by using the controller5, the user can perform a game operation of moving the controller5itself, in addition to the conventional typical game operation of pressing the operation buttons. For example, it enables an operation of tilting the controller5to an intended attitude, an operation of specifying an intended position on the screen with the controller5, an operation of moving the controller5itself, etc. while the controller5does not include the display means for displaying the game image in the present embodiment, it may include display means for displaying, for example, an image representing the battery level, etc.

[4. Configuration of Terminal Device7]

Next, a configuration of the terminal device7will be described with reference toFIGS. 8 to 10.FIG. 8is a diagram showing an external configuration of the terminal device7. InFIG. 8, (a) is a front view of the terminal device7, (b) is a top view thereof, (c) is a right side view thereof, and (d) is a bottom view thereof.FIG. 9is a diagram showing the terminal device7being held by the user.

As shown inFIG. 8, the terminal device7includes a housing50generally in a horizontally-elongated rectangular plate shape. The housing50has such a size that it can be held by the user. Thus, the user can hold and move the terminal device7, and can change the position where the terminal device7is placed. The terminal device7includes the LCD51on the surface of the housing50. The LCD51is provided near the center of the surface of the housing50. Therefore, the user can hold and move the terminal device while looking at the screen of the LCD51by holding opposing end portions of the housing50with respect to the LCD51, as shown inFIG. 9. Note that whileFIG. 9shows an example where the user holds the terminal device7in a horizontal position (in a horizontally-oriented direction) by holding left and right opposing end portions of the housing50with respect to the LCD51, the user can hold the terminal device7in a vertical position (in a vertically-oriented direction).

As shown in (a) ofFIG. 8, the terminal device7includes the touch panel52on the screen of the LCD51as operation means. In the present embodiment, the touch panel52is a resistive-type touch panel. Note however that the touch panel is not limited to those of the resistive type, but may be a touch panel of any type including, for example, a capacitive type, etc. The touch panel52may be of a single-touch type or a multi-touch type. In the present embodiment, a touch panel having the same resolution (detection precision) as the resolution of the LCD51is used as the touch panel52. Note however that the resolution of the touch panel52does not always need to coincide with the resolution of the LCD51. While a stylus is usually used for making an input on the touch panel52, the present invention is not limited to a stylus, and an input may be made on the touch panel52with a finger of the user. Note that the housing50may be provided with a hole for accommodating a stylus used for performing an operation on the touch panel52. Thus, since the terminal device7includes the touch panel52, the user can operate the touch panel52while moving the terminal device7. That is, the user can move the screen of the LCD51while directly (by means of the touch panel52) making an input on the screen.

As shown inFIG. 8, the terminal device7includes two analog sticks53A and53B and a plurality of buttons54A to54L, as operation means. The analog sticks53A and53B are each a direction-specifying device. The analog sticks53A and53B are each configured so that the stick portion operated with a finger of the user can be slid or tilted in any direction (at any angle in the upper, lower, left, right and diagonal directions) with respect to the surface of the housing50. The left analog stick53A is provided on the left side of the screen of the LCD51, and the right analog stick53B is provided on the right side of the screen of the LCD51. Therefore, the user can make a direction-specifying input by using an analog stick with either the left or the right hand. As shown inFIG. 9, the analog sticks53A and53B are provided at such positions that the user can operate them while holding the left and right portions of the terminal device7, and therefore the user can easily operate the analog sticks53A and53B even when holding and moving the terminal device7.

The buttons54A to54L are each operation means for making a predetermined input. As will be illustrated below, the buttons54A to54L are provided at such positions that the user can operate them while holding the left and right portions of the terminal device7(seeFIG. 9). Therefore, the user can easily operate these operation means even when holding and moving the terminal device7.

As shown in (a) ofFIG. 8, the cross button (direction-input button)54A and the buttons54B to54H, of the operation buttons54A to54L, are provided on the front surface of the housing50. That is, these buttons54A to54H are provided at such positions that they can be operated by the thumbs of the user (seeFIG. 9).

The cross button54A is provided on the left side of the LCD51and under the left analog stick53A. That is, the cross button54A is provided at such a position that it can be operated with the left hand of the user. The cross button54A has a cross shape, and is a button with which it is possible to specify upper, lower, left and right directions. The buttons54B to54D are provided on the lower side of the LCD51. These three buttons54B to54D are provided at such positions that they can be operated with either the left or the right hand. The four buttons54E to54H are provided on the right side of the LCD51and under the right analog stick53B. That is, the four buttons54E to54H are provided at such positions that they can be operated with the right hand of the user. Moreover, the four buttons54E to54H are provided on the upper, lower, left and right side (of the center position among the four buttons54E to54H). Therefore, with the terminal device7, the four buttons54E to54H can also serve as buttons with which the user specifies the upper, lower, left and right directions.

As shown in (a), (b) and (c) ofFIG. 8, the first L button54I and the first R button54J are provided in upper corner portions of the housing50(the upper left portion and the upper right portion). Specifically, the first L button54I is provided at the left end of the upper side surface of the plate-like housing50so that it is exposed on the upper and left side surfaces. The first R button54J is provided at the right end of the upper side surface of the housing50so that it is exposed on the upper and right side surfaces. Thus, the first L button54I is provided at such a position that it can be operated with the left index finger of the user, and the first R button54J is provided at such a position that it can be operated with the right index finger of the user (seeFIG. 9).

As shown in (b) and (c) ofFIG. 8, the second L button54K and the second R button54L are provided on leg portions59A and59B protruding from the back surface of the plate-like housing50(i.e., the surface opposite to the front surface where the LCD51is provided). Specifically, the second L button54K is provided slightly toward the upper side in the left portion (the left portion as viewed from the front surface side) of the back surface of the housing50, and the second R button54I is provided slightly toward the upper side in the right portion (the right portion as viewed from the front surface side) of the back surface of the housing50. In other words, the second L button54K is provided generally on the reverse side of the left analog stick53A provided on the front surface, and the second R button54L is provided generally on the reverse side of the right analog stick53B provided on the front surface. Thus, the second L button54K is provided at such a position that it can be operated with the left middle finger of the user, and the second R button54L is provided at such a position that it can be operated with the right middle finger of the user (seeFIG. 9). The second L button54K and the second R button54L are provided on the diagonally-upwardly-facing surfaces of the leg portions59A and59B, and have diagonally-upwardly-facing button surfaces, as shown in (c) ofFIG. 8. It is believed that the middle fingers move in the up/down direction when the user holds the terminal device7, and it will be easier for the user to press the second L button54K and the second R button54L if the button surfaces are facing upward. The provision of the leg portions on the back surface of the housing50makes it easier for the user to hold the housing50, and the provision of the buttons on the leg portions makes it easier for the user to make an operation while holding the housing50.

Note that with the terminal device7shown inFIG. 8, since the second1, button54K and the second R button54L are provided on the back surface, when the terminal device7is put down with the screen of the LCD51(the front surface of the housing50) facing up, the screen may not lie completely horizontal. Therefore, in other embodiments, three or more leg portions may be provided on the back surface of the housing50. Then, it can be put down on the floor surface with the leg portions in contact with the floor surface with the screen of the LCD51facing up, and it is therefore possible to put down the terminal device7so that the screen lies horizontal. A detatchable leg portion may be added so that the terminal device7can be put down horizontally.

The buttons54A to54L are each assigned a function in accordance with the game program. For example, the cross button54A and the buttons54E to54H may be used for a direction-specifying operation, a selection operation, etc., whereas the buttons54B to54E may be used for the OK operation, the cancel operation, etc.

Note that although not shown in the figures, the terminal device7may include a power button for turning ON/OFF the power of the terminal device7. The terminal device7may include a button for turning ON/OFF the display of the screen of the LCD51, a button for performing a connection setting (pairing) with the game device3, and a button for adjusting the volume of the speaker (a speaker67shown inFIG. 10).

As shown in (a) ofFIG. 8, the terminal device7includes a marker section including a marker55A and a marker55B (the marker section55shown inFIG. 10) on the front surface of the housing50. The marker section55is provided on the upper side of the LCD51. The marker55A and the marker55B are each formed by one or more infrared LED, as are the markers6R and6L of the marker device6. The marker section55is used for the game device3to calculate the movement, etc., of the controller5, as is the marker device6described above. The game device3can control the lighting of the infrared LEDs of the marker section55.

The terminal device7includes the camera56as image-capturing means. The camera56includes an image-capturing element (e.g., a CCD image sensor, a CMOS image sensor, or the like) having a predetermined resolution, and a lens. As shown inFIG. 8, the camera56is provided on the front surface of the housing50in the present embodiment. Therefore, the camera56can capture an image of the face of the user holding the terminal device7, and can capture an image of the user playing a game while looking at the LCD51, for example.

Note that the terminal device7includes a microphone (a microphone69shown inFIG. 10) as sound input means. A microphone hole60is provided on the front surface of the housing50. The microphone69is provided inside the housing50behind the microphone hole60. The microphone detects the sound around the terminal device7such as the voice of the user.

The terminal device7includes a speaker (the speaker67shown inFIG. 10) as sound outputting means. As shown in (d) ofFIG. 8, a speaker hole57is provided on the lower side surface of the housing50. The output sound from the speaker67is outputted from the speaker hole57. In the present embodiment, the terminal device7includes two speakers, and the speaker hole57is provided at the position of each of the left speaker and the right speaker.

The terminal device7includes an extension connector58via which another device can be connected to the terminal device7. In the present embodiment, the extension connector58is provided on the lower side surface of the housing50as shown in ofFIG. 8. Note that the other device connected to the extension connector58may be any device, and may be for example a game-specific controller (gun-shaped controller, etc.) or an input device such as a keyboard. The extension connector58may be absent if there is no need to connect another device.

Note that with the terminal device7shown inFIG. 8, the shape of each operation button, the shape of the housing50, the number and the positions of the components, etc., are merely illustrative, and the present invention can be realized with other shapes, numbers, and positions.

Next, an internal configuration of the terminal device7will be described with reference toFIG. 10.FIG. 10is a block diagram showing an internal configuration of the terminal device7. As shown inFIG. 10, in addition to the configuration shown inFIG. 8, the terminal device7includes a touch panel controller61, a magnetic sensor62, the acceleration sensor63, the gyrosensor64, a user interface controller (UI controller)65, a codec LSI66, the speaker67, a sound IC68, the microphone69, a wireless module70, an antenna71, an infrared communication module72, a flash memory73, a power supply IC74, and a battery75. These electronic components are mounted on an electronic circuit board and accommodated in the housing50.

The UI controller65is a circuit for controlling the input/output of data to/from various types of input/output sections. The UI controller65is connected to the touch panel controller61, an analog stick53(the analog sticks53A and53B), an operation button54(the operation buttons54A to54L), the marker section55, the magnetic sensor62, the acceleration sensor63, and the gyrosensor64. The UI controller65is connected to the codec LSI66and the extension connector58. The power supply IC74is connected to the UT controller65, and power is supplied to various sections via the UI controller65. The built-in battery75is connected to the power supply IC74to supply power. A charger76or a cable with which power can be obtained from an external power source can be connected to the power supply IC74via a connector, or the like, and the terminal device7can receive power supply from or be charged by an external power source using the charger76or the cable. Note that the terminal device7may be charged by attaching the terminal device7to a cradle (not shown) having a charging function.

The touch panel controller61is a circuit connected to the touch panel52for controlling the touch panel52. The touch panel controller61generates touch position data of a predetermined format based on a signal from the touch panel52, and outputs it to the UI controller65. The touch position data represents the coordinates of a position on the input surface of the touch panel52at which an input is made. Note that the touch panel controller61reads a signal from the touch panel52and generates touch position data at a rate of once per a predetermined amount of time. Various control instructions for the touch panel52are outputted from the UI controller65to the touch panel controller61.

The analog stick53outputs, to the UI controller65, stick data representing the direction and the amount of side (or tilt) of the stick portion operated with a finger of the user. The operation button54outputs, to the UI controller65, operation button data representing the input status of each of the operation buttons54A to54L (whether it is pressed).

The magnetic sensor62detects the azimuthal direction by sensing the size and direction of the magnetic field. Azimuthal direction data representing the detected azimuthal direction is outputted to the UI controller65. Control instructions for the magnetic sensor62are outputted from the UI controller65to the magnetic sensor62. While there are sensors using an MI (magnetic impedance) element, a fluxgate sensor, a Hail element, a GMR (giant magneto-resistive) element, a TMR (tunnel magneto-resistance) element, an AMR (anisotropic magneto-resistive) element, etc., the magnetic sensor62may be any sensor as long as it is possible to detect the azimuthal direction. Note that strictly speaking, in a place where there is a magnetic field other than the geomagnetic field, the obtained azimuthal direction data does not represent the azimuthal direction. Nevertheless, if the terminal device7moves, the azimuthal direction data changes, and it is therefore possible to calculate the change in the attitude of the terminal device7.

The acceleration sensor63is provided inside the housing50for detecting the magnitude of the linear acceleration along each of the directions of the three axes (the x, y and z axes shown in (a) ofFIG. 8). Specifically, the acceleration sensor63detects the magnitude of the linear acceleration along each of the axes, where the x axis lies in the longitudinal direction of the housing50, the y axis lies in the width direction of the housing50, and the z axis lies in the direction vertical to the surface of the housing50. Acceleration data representing the detected acceleration is outputted to the UI controller65. Control instructions for the acceleration sensor63are outputted from the UI controller65to the acceleration sensor63. While the acceleration sensor63is assumed to be a capacitive-type MEMS-type acceleration sensor, for example, in the present embodiment, other types of acceleration sensors may be employed in other embodiments. The acceleration sensor63may be an acceleration sensor for 1-axis or 2-axis detection.

The gyrosensor64is provided inside the housing50for detecting angular velocities about the three axes, i.e., the x axis, the y axis and the z axis. Angular velocity data representing the detected angular velocities is outputted to the UI controller65. Control instructions for the gyrosensor64are outputted from the UI controller65to the gyrosensor64. Note that the number and combination of gyrosensors used for detecting angular velocities about three axes may be any number and combination, and the gyrosensor64may be formed by a 2-axis gyrosensor and a 1-axis gyrosensor, as is the gyrosensor48. The gyrosensor64may be a gyrosensor for 1-axis or 2-axis detection.

The UI controller65outputs, to the codec LSI66, operation data including touch position data, stick data, operation button data, azimuthal direction data, acceleration data, and angular velocity data received from various components described above. Note that if another device is connected to the terminal device7via the extension connector58, data representing an operation performed on the other device may be further included in the operation data.

The codec LSI66is a circuit for performing a compression process on data to be transmitted to the game device3, and an expansion process on data transmitted from the game device3. The LCD51, the camera56, the sound IC68, the wireless module70, the flash memory73, and the infrared communication module72are connected to the codec LSI66. The codec LSI66includes a CPU77and an internal memory78. While the terminal device7does not perform the game process itself, the terminal device7needs to execute a minimal program for the management thereof and for the communication. The terminal device7is started up, when a program stored in the flash memory73is read out to the internal memory78and executed by the CPU77upon power-up. Some area of the internal memory78is used as the VRAM for the LCD51.

The camera56captures an image in response to an instruction from the game device3, and outputs the captured image data to the codec LSI66. Control instructions for the camera56, such as an image-capturing instruction, are outputted from the codec LSI66to the camera56. Note that the camera56can also record a video. That is, the camera56can repeatedly capture images and repeatedly output the image data to the codec LSI66.

The sound IC68is a circuit connected to the speaker67and the microphone69for controlling input/output of sound data to/from the speaker67and the microphone69. That is, when sound data is received from the codec LSI66, the sound IC68outputs a sound signal obtained by performing D/A conversion on the sound data to the speaker67so that a sound is outputted from the speaker67. The microphone69detects the sound propagated to the terminal device7(the sound of the user, etc.), and outputs a sound signal representing such a sound to the sound IC68. The sound IC68performs A/D conversion on the sound signal from the microphone69to output sound data of a predetermined format to the codec LSI66.

The codec LSI66transmits, as terminal operation data, image data from the camera56, sound data from the microphone69and operation data from the UI controller65to the game device3via the wireless module70. In the present embodiment, the codec LSI66performs a compression process similar to that of the codec LSI27on the image data and the sound data. The terminal operation data and the compressed image data and sound data are outputted, as transmit data, to the wireless module70. The antenna71is connected to the wireless module70, and the wireless module70transmits the transmit data to the game device3via the antenna71. The wireless module70has a similar function to that of the terminal communication module28of the game device3. That is, the wireless module70has a function of connecting to a wireless LAN by a scheme in conformity with the IEEE802.11n standard, for example. The transmitted data may be encrypted as necessary or may not be encrypted.

As described above, the transmit data transmitted from the terminal device7to the game device3includes the operation data (the terminal operation data), the image data, and the sound data. Note that in a case where another device is connected to the terminal device7via the extension connector58, data received from the other device may be further included in the transmit data. The infrared communication module72establishes infrared communication in conformity with the IRDA standard, for example, with the other device. The codec LSI66may transmit, to the game device3, data received via the infrared communication while it is included in the transmit data as necessary.

As described above, the compressed image data and sound data are transmitted from the game device3to the terminal device7. These data are received by the codec LSI66via the antenna71and the wireless module70. The codec LSI66expands the received image data and sound data. The expanded image data is outputted to the LCD51, and an image is displayed on the LCD51. The expanded sound data is outputted to the sound IC68, and the sound IC68outputs the sound from the speaker67.

In a case where control data is included in data received from the game device3, the codec LSI66and the UI controller65give control instructions to various sections in accordance with the control data. As described above, the control data is data representing control instructions for the components of the terminal device7(the camera56, the touch panel controller61, the marker section55, sensors62to64, and the infrared communication module72in the present embodiment). In the present embodiment, control instructions represented by control data may be instructions to activate the operation of the components or inactivate (stop) the operation thereof. That is, components that are not used in the game may be inactivated in order to reduce the power consumption, in which case it is ensured that data from the inactivated components are not included in the transmit data transmitted from the terminal device7to the game device3. Note that for the marker section55, which is an infrared LED, the control can be done simply by turning ON/OFF the power supply thereto.

While the terminal device7includes operation means such as the touch panel52, the analog stick53and the operation button54, as described above, other operation means may be included instead of, or in addition to, these operation means in other embodiments.

While the terminal device7includes the magnetic sensor62, the acceleration sensor63and the gyrosensor64as sensors for calculating the movement of the terminal device7(including the position and the attitude thereof, or changes in the position and the attitude thereof), it may only include one or two of these sensors in other embodiments. In other embodiments, other sensors may be included instead of, or in addition to, these sensors.

While the terminal device7includes the camera56and the microphone69, it may not include the camera56and the microphone69or it may include only one of them in other embodiments.

While the terminal device7includes the marker section55as a configuration for calculating the positional relationship between the terminal device7and the controller5(the position and/or attitude, etc., of the terminal device7as seen from the controller5), it may not include the marker section55in other embodiments. In other embodiments, the terminal device7may include other means as a configuration for calculating the positional relationship. For example, in other embodiments, the controller5may include a marker section, and the terminal device7may include an image-capturing element. Moreover, in such a case, the marker device6may include an image-capturing element, instead of an infrared LED.

Next, the details of the game process performed in the present game system will be described. First, various data used in the game process will be described.FIG. 11is a table showing various data used in the game process.FIG. 11is a table showing primary data to be stored in the main memory (the external main memory12or the internal main memory11e) of the game device3. As shown inFIG. 11, the main memory of the game device3stores a game program90, receive data91, and process data106. Note that in addition to those shown inFIG. 11, the main memory also stores data necessary for the game, such as image data of various objects appearing in the game, and sound data used in the game, etc.

At an appropriate point in time after the power of the game device3is turned ON, a part or whole of the game program90is loaded from the optical disc4and stored in the main memory. Note that the game program90may be obtained from the flash memory17or an external device of the game device3(e.g., via the Internet), instead of from the optical disc4. A part of the game program90(e.g., a program for calculating the attitude of the controller5and/or the terminal device7) may be pre-stored in the game device3.

The receive data91are various data received from the controller5and the terminal device7. The receive data91includes controller operation data92, terminal operation data97, camera image data104, and microphone sound data105. If a plurality of controllers5are connected, there are a plurality of controller operation data92. If a plurality of terminal devices7are connected, there are a plurality of terminal operation data97, a plurality of camera image data104, and a plurality of microphone sound data105.

The controller operation data92is data representing an operation performed by the user (player) on the controller5. The controller operation data92is transmitted from the controller5to be obtained by the game device3and stored in the main memory. The controller operation data92includes first operation button data93, first acceleration data94, first angular velocity data95, and marker coordinate data96. Note that the main memory may store a predetermined number of latest (lastly obtained) sets of controller operation data.

The first operation button data93is data representing the input status of the operation buttons32ato32iprovided on the controller5. Specifically, the first operation button data93represents whether each of the operation buttons32ato32iis pressed.

The first acceleration data94is data representing the acceleration (acceleration vector) detected by the acceleration sensor37of the controller5. While the first acceleration data94herein represents three-dimensional acceleration of which each component is the acceleration for one of the three axis directions of X, Y and Z shown inFIG. 3, it may represent acceleration for any one or more direction in other embodiments.

The first angular velocity data95is data representing the angular velocity detected by the gyrosensor48in the controller5. While the first angular velocity data95herein represents angular velocity about each of the three axes of X, Y and Z shown inFIG. 3, it may represent angular velocity about any one or more axis in other embodiments.

The marker coordinate data96is data representing coordinates calculated by the image processing circuit41of the image capturing/processing section35, i.e., the marker coordinates described above. The marker coordinates are expressed in a two-dimensional coordinate system for representing a position on a plane corresponding to the captured image, and the marker coordinate data96represents coordinate values in the two-dimensional coordinate system.

Note that the controller operation data92may be data representing an operation by the user operating the controller5, and may be data including only some of the data93to96. In a case where the controller5includes other input means (e.g., a touch panel or an analog stick, etc.), the controller operation data92may include data representing an operation performed on the other input means. Note that in a case where the movement of the controller5itself is used as a game operation as in the present embodiment, the controller operation data92includes data whose value changes in accordance with the movement of the controller5itself, as is the first acceleration data94, the first angular velocity data95or the marker coordinate data96.

The terminal operation data97is data representing an operation performed by the user on the terminal device7. The terminal operation data97is transmitted from the terminal device7and obtained by the game device3to be stored in the main memory. The terminal operation data97includes second operation button data98, stick data99, touch position data100, second acceleration data101, second angular velocity data102, and azimuthal direction data. Note that the main memory may store a predetermined number of latest (lastly obtained) sets of terminal operation data.

The second operation button data98is data representing the input status of the operation buttons54A to54L provided on the terminal device7. Specifically, the second operation button data98represents whether each of the operation buttons54A to54L is pressed.

The stick data99is data representing the direction and the amount of slide (or tilt) of the stick portion of the analog stick53(the analog sticks53A and53B). The direction and the amount may be represented as two-dimensional coordinates or a two-dimensional vector, for example.

The touch position data100is data representing the position (touch position) on the input surface of the touch panel52at which an input is made. In the present embodiment, the touch position data100represents coordinate values in a two-dimensional coordinate system for representing a position on the input surface. Note that in a case where the touch panel52is of a multi-touch type, the touch position data100may represent a plurality of touch positions.

The second acceleration data101is data representing the acceleration (acceleration vector) detected by the acceleration sensor63. While the second acceleration data101represents three-dimensional acceleration of which each component is the acceleration for one of the three axes of x, y and z shown inFIG. 8in the present embodiment, it may represent acceleration for any one or more direction in other embodiments.

The second angular velocity data102is data representing the angular velocity detected by the gyrosensor64. While the second angular velocity data102represents angular velocity about each of the three axis directions of x, y and z shown inFIG. 8in the present embodiment, it may represent angular velocity about any one or more axis in other embodiments.

Azimuthal direction data103is data representing the azimuthal direction detected by the magnetic sensor62. In the present embodiment, the azimuthal direction data103represents the direction of a predetermined azimuthal direction (e.g., north) with respect to the terminal device7. Note however that in a place where there is a magnetic field other than the geomagnetic field, the azimuthal direction data103does not strictly represent the absolute azimuthal direction (e.g., north). Nevertheless, it represents a relative direction of the terminal device7with respect to the direction of the magnetic field in that place, and it is therefore possible to calculate the change in the attitude of the terminal device7even in such cases.

Note that the terminal operation data97may be data representing an operation performed by the user on the terminal device7, and may be data including only one of the data98to103described above. In a case where the terminal device7includes other input means (e.g., a touch pad, image-capturing means of the controller5, etc.), the terminal operation data97may include data representing an operation performed on the other input means. Note that in a case where the movement of the terminal device7itself is used as a game operation as in the present embodiment, the terminal operation data97includes data whose value changes in accordance with the movement of the terminal device7itself, as is the second acceleration data101, the second angular velocity data102or the azimuthal direction data103.

The camera image data104is data representing the image (camera image) captured by the camera56of the terminal device7. The camera image data104is image data obtained by the codec LSI27expanding the compressed image data from the terminal device7, and the data is stored in the main memory by the input/output processor11a. Note that the main memory may store a predetermined number of latest (lastly obtained) sets of camera image data.

The microphone sound data105is data representing the sound (microphone sound) detected by the microphone69of the terminal device7. The microphone sound data105is sound data obtained by the codec LSI27expanding the compressed sound data transmitted from the terminal device7, and the data is stored in the main memory by the input/output processor11a.

The process data106is data used in the game process (FIG. 12) to be described later. The process data106includes control data107, controller attitude data108, terminal attitude data109, image recognition data110, and sound recognition data111. Note that in addition to those shown inFIG. 11, the process data106also includes various data used in the game process, such as data representing various parameters set for various objects appearing in the game.

The control data107is data representing control instructions for the components of the terminal device7. For example, the control data107represents an instruction for controlling the lighting of the marker section55, an instruction for controlling the image-capturing operation of the camera56, etc. The control data107is transmitted to the terminal device7at an appropriate point in time.

The controller attitude data108is data representing the attitude of the controller5. In the present embodiment, the controller attitude data108is calculated based on the first acceleration data94, the first angular velocity data95and the marker coordinate data96included in the controller operation data92. The method for calculating the controller attitude data108will be described later in step S23.

The terminal attitude data109is data representing the attitude of the terminal device7. In the present embodiment, the terminal attitude data109is calculated based on the second acceleration data101, the second angular velocity data102and the azimuthal direction data103included in the terminal operation data97. The method for calculating the terminal attitude data109will be described later in step S24.

The image recognition data110is data representing the results of a predetermined image recognition process for the camera image. The image recognition process may be any process as long as it detects any feature of the camera image to output the results of the detection, and may be, for example, a process of extracting a predetermined object (e.g., the face of the user, a marker, etc.) from the camera image and calculating information regarding the extracted object.

The sound recognition data111is data representing the results of a predetermined sound recognition process for the microphone sound. The sound recognition process may be any process as long as it detects any feature from the microphone sound to output the results of the detection, and may be, for example, a process of detecting words of the user or a process of simply outputting the sound volume.

Next, with reference toFIG. 12, the details of the game process performed by the game device3will be described.FIG. 12is a main flow chart showing the flow of the game process performed by the game device3. When the power of the game device3is turned ON, the CPU10of the game device3executes a boot program stored in a boot ROM not shown, so as to initialize each unit, including the main memory. Then, the game program stored in the optical disc4is loaded to the main memory, and the CPU10starts executing the game program. Note that the game device3may be configured so as to execute the game program stored in the optical disc4immediately after power-up, or it may be configured so that a built-in program is executed after power-up for displaying a predetermined menu screen first, and then the game program stored in the optical disc4is executed when the start of the game is instructed by the user. The flowchart ofFIG. 12is a flowchart showing the process to be performed after processes described above are completed.

Note that the process of the steps of the flowchart shown inFIG. 12is merely illustrative, and the order of steps to be performed may be switched around as long as similar results are obtained. The values of the variables, and the threshold values used in determination steps are also merely illustrative, and other values may be used as necessary. While the present embodiment is described while assuming that the processes of the steps of the flow chart are performed by the CPU10, processes of some of the steps may be performed by a processor or a dedicated circuit other than the CPU10.

First, in step S1, the CPU10performs an initialization process. The initialization process is, for example, a process of constructing a virtual game space, placing objects appearing in the game space at their initial positions, and setting initial values of various parameters used in the game process.

In the present embodiment, in the initialization process, the CPU10controls the lighting of the marker device6and the marker section55based on the type of the game program. Here, the game system1has two image-capturing objects for the image-capturing means of the controller5(the image capturing/processing section35), i.e., the marker device6and the marker section55of the terminal device7. Either or both of the marker device6and the marker section55may be used, depending on the content of the game (the type of the game program). Note that the game program90includes data representing whether each of the marker device6and the marker section55should be lit. The CPU10reads out this data to determine whether or not to light them. When lighting the marker device6and/or the marker section55, the following process is performed.

That is, when lighting the marker device6, the CPU10transmits, to the marker device6, a control signal instructing to light the infrared LEDs of the marker device6. The transmission of the control signal may be simply supplying the power. In response to this, the infrared LEDs of the marker device6are lit. On the other hand, when lighting the marker section55, the CPU10generates control data representing an instruction for lighting the marker section55and stores the data in the main memory. The generated control data is transmitted to the terminal device7in step S10to be described later. The control data received by the wireless module70of the terminal device7is sent to the UI controller65via the codec LSI66, and the UI controller65gives a lighting instruction to the marker section55. This lights the infrared LEDs of the marker section55. Note that while a case where the marker device6and the marker section55are lit has been described above, the marker device6and the marker section55can be turned off through a similar process to the process of lighting them.

The process of step S2is performed, following step S1described above. Thereafter, the process loop including a series of processes of steps S2to S11is repeatedly performed at a rate of once per a predetermined amount of time one frame period).

In step S2, the CPU10obtains controller operation data transmitted from the controller5. Since the controller5repeatedly transmits the controller operation data to the game device3, the controller operation data is successively received by the controller communication module19in the game device3, and the received controller operation data is successively stored in the main memory by the input/output processor11a. The transmission/reception interval is preferably shorter than the game process time, and is, for example, 1/200 sec. In step S2, the CPU10reads out the latest controller operation data92from the main memory. The process of step S3is performed, following step S2.

In step S3, the CPU10obtains various data transmitted from the terminal device7. Since the terminal device7repeatedly transmits the terminal operation data, the camera image data and the microphone sound data to the game device3, the game device3successively receives these data. In the game device3, the terminal communication module28successively receives these data, and the camera image data and the microphone sound data are successively expanded by the codec LSI27. Then, the input/output processor11asuccessively stores the terminal operation data, the camera image data and the microphone sound data in the main memory. In step S3, the CPU10reads out the latest terminal operation data97from the main memory. The process of step S4is performed, following step S3.

In step S4, the CPU10performs the game control process. The game control process is a process for allowing the game to progress by, for example, performing processes such as controlling the action of an object in the game space in accordance with the game operation by the user. In the present embodiment, the user can play various games by using the controller5and/or the terminal device7. Now, with reference toFIG. 13, the game control process will be described.

FIG. 13is a flow chart showing the detailed flow of the game control process. Note that while the series of processes shown inFIG. 13are various processes that can be performed in a case where the controller5and the terminal device7are used as controller devices, it is not necessary to perform all of the processes, and only some of the processes may be performed depending on the type and content of the game.

In the game control process, first, in step S21, the CPU10determines whether or not to change the marker to be used. In the present embodiment, the process of controlling the lighting of the marker device6and the marker section55is performed at the start of the game process (step S1), as described above. Here, depending on the game, a marker or markers to be used (lit) among the marker device6and the marker section55may be changed in the middle of the game. It may be possible to use both the marker device6and the marker section55depending on the game, but if they are both lit, one of the markers may be erroneously detected for the other marker. Therefore, there are cases where it is preferred to switch the lighting during the game so that only one of them is lit. In view of such cases, the process of step S21is a process of determining whether the object to be lit is changed in the middle of the game.

The determination of step S21can be made by the following method, for example. That is, the CPU10can make the determination based on whether the game status (the stage of the game, the object to be controlled, etc.) has changed. When the game status changes, the control method may be switched between a control method in which the controller5is controlled facing the marker device6, and a control method in which the controller5is controlled facing the marker section55. The CPU10can make the determination based on the attitude of the controller5. That is, the determination can be made based on whether the controller5is facing the marker device6or facing the marker section55. Note that the attitude of the controller5can be calculated based on the detection results of the acceleration sensor37and the gyrosensor48, for example (see step S23to be described later). The CPU10can make the determination based on whether there has been an instruction of change from the user.

If the determination result of step S21is affirmative, the process of step S22is performed. On the other hand, if the determination result of step S21is negative, the process of step S22is skipped and the process of step S23is performed.

In step S22, the CPU10controls the lighting of the marker device6and the marker section55. That is, it changes the lighting status of the marker device6and/or the marker section55. Note that the specific process of lighting or turning off the marker device6and/or the marker section55can be performed in a similar manner to that of step S1. The process of step S23is performed, following step S22.

As described above, according to the present embodiment, the light emission (lighting) of the marker device6and the marker section55can be controlled depending on the type of the game program through the process of step S1, and the light emission (lighting) of the marker device6and the marker section55can be controlled in accordance with the game status through the process of steps S21and S22.

In step S23, the CPU10calculates the attitude of the controller5. In the present embodiment, the attitude of the controller5is calculated based on the first acceleration data94, the first angular velocity data95and the marker coordinate data96. The method for calculating the attitude of the controller5will now be described.

First, the CPU10calculates the attitude of the controller5based on the first angular velocity data95stored in the main memory. While the method for calculating the attitude of the controller5from the angular velocity may be any method, the attitude is calculated using the previous attitude (the attitude calculated in the previous iteration) and the current angular velocity (the angular velocity obtained in step S2in the current iteration of the process loop). Specifically, the CPU10calculates the attitude by rotating the previous attitude by a unit time's worth of the current angular velocity. Note that the previous attitude is represented by the controller attitude data108stored in the main memory, and the current angular velocity is represented by the first angular velocity data95stored in the main memory. Therefore, the CPU10reads out the controller attitude data108and the first angular velocity data95from the main memory to calculate the attitude of the controller5. The data representing “the attitude based on the angular velocity” calculated as described above is stored in the main memory.

Note that where the attitude is calculated from the angular velocity, it is preferred that an initial attitude is set. That is, where the attitude of the controller5is calculated from the angular velocity, the CPU10initially calculates the initial attitude of the controller5. The initial attitude of the controller5may be calculated based on the acceleration data, or the player may be prompted to perform a predetermined operation with the controller5in a particular attitude so that the particular attitude at the point in time when the predetermined operation is performed is used as the initial attitude. Note that while it is preferred to calculate the initial attitude in a case where the attitude of the controller5is calculated as an absolute attitude with respect to a predetermined direction in the space, the initial attitude may not be calculated in a case where the attitude of the controller5is calculated as a relative attitude with respect to the attitude of the controller5at the start of the game, for example.

Next, the CPU10corrects the attitude of the controller5calculated based on the angular velocity by using the first acceleration data94. Specifically, the CPU10first reads out the first acceleration data94from the main memory and calculates the attitude of the controller5based on the first acceleration data94. Here, in a state where the controller5is substantially stationary, the acceleration acting upon the controller5means the gravitational acceleration. Therefore, in this state, the direction of the gravitational acceleration (the direction of gravity) can be calculated by using the first acceleration data94outputted from the acceleration sensor37, and it is therefore possible to calculate the direction (attitude) of the controller5with respect to the direction of gravity based on the first acceleration data94. The data representing “the attitude based on the acceleration” calculated as described above is stored in the main memory.

After calculating the attitude based on the acceleration, the CPU10then corrects the attitude based on the angular velocity by using the attitude based on the acceleration. Specifically, the CPU10reads out data representing the attitude based on the angular velocity and data representing the attitude based on the acceleration from the main memory, and makes a correction such that the attitude based on the angular velocity data is brought closer to the attitude based on the acceleration data at a predetermined rate. The predetermined rate may be a predetermined fixed value, and may be set in accordance with the acceleration represented by the first acceleration data94, etc. With the attitude based on the acceleration, the attitude cannot be calculated for the rotation direction about the direction of gravity, and therefore the CPU10may not make a correction for the rotation direction. In the present embodiment, data representing the corrected attitude obtained as described above is stored in the main memory.

After the attitude based on the angular velocity is corrected as described above, the CPU10further corrects the corrected attitude using the marker coordinate data96. First, the CPU10calculates the attitude of the controller5based on the marker coordinate data96(the attitude based on marker coordinates). Since the marker coordinate data96represents positions of the markers6R and6L within the captured image, the attitude of the controller5can be calculated for the roll direction (the rotation direction about the Z axis) can be calculated from these positions. That is, the attitude of the controller5for the roll direction can be calculated from the gradient of the straight line connecting between the position of the marker6R and the position of the marker6L within the captured image. In a case where the position of the controller5with respect to the marker device6can be identified (e.g., a case where it can be assumed that the controller5is located in front of the marker device6), the attitude of the controller5for the pitch direction and that for the yaw direction can be calculated from the position of the marker device6within the captured image. For example, when the positions of the markers6R and6L move to the left within the captured image, it can be determined that the controller5has changed its orientation (attitude) to the right. Thus, the attitude of the controller5for the pitch direction and that for the yaw direction can be calculated from the positions of the marker6R and the marker6l. As described above, it is possible to calculate the attitude of the controller5based on the marker coordinate data96.

After the attitude based on marker coordinates is calculated, the CPU10next corrects the corrected attitude (the attitude which has been corrected by the attitude based on the acceleration) by the attitude based on marker coordinates. That is, the CPU10makes a correction such that the corrected attitude is brought closer to the attitude based on marker coordinates at a predetermined rate. The predetermined rate may be a predetermined fixed value. The correction by the attitude based on marker coordinates may be made only for any one or two of the roll direction, the pitch direction and the yaw direction. For example, where the marker coordinate data96is used, since it is possible to calculate the attitude with high precision for the roll direction, the CPU10may make the correction using the attitude based on the marker coordinate data96only for the roll direction. If the image-capturing element40of the controller5does not capture the image of the marker device6or the marker section55, it is not possible to calculate the attitude based on the marker coordinate data96, and therefore the correction process using the marker coordinate data96may not be performed in such a case.

In the above description, the CPU10corrects the first attitude of the controller5calculated based on the first angular velocity data95using the first acceleration data94and the marker coordinate data96. Here, with the method using the angular velocity, among the methods for calculating the attitude of the controller5, it is possible to calculate the attitude no matter how the controller5is moving. On the other hand, with the method using the angular velocity, since the attitude is calculated by cumulatively adding the successively-detected angular velocities, accumulation of errors, or the like, may lead to poor precision, and a so-called “temperature drift” problem may deteriorate the precision of the gyrosensor. With the method using the acceleration, errors do not accumulate, but it is not possible to calculate the attitude with high precision in a state where the controller5is being moved violently (since the direction of gravity cannot be detected accurately). With the method using marker coordinates, the attitude can be calculated with high precision (particularly for the roll direction), but it is not possible to calculate the attitude in a state where it is not possible to capture an image of the marker section55. As opposed to this, the attitude of the controller5can be calculated more accurately in the present embodiment since three different methods with different characteristics are used as described above. Note that in other embodiments, the attitude may be calculated by using any one or two of the three methods described above. Where the lighting of the markers is controlled in the process of step S1or S22, it is preferred that the CPU10calculates the attitude of the controller5using at least marker coordinate.

The process of step S24is performed, following step S23. In step S24, the CPU10calculates the attitude of the terminal device7. That is, since the terminal operation data97obtained from the terminal device7includes the second acceleration data101, the second angular velocity data102, and the azimuthal direction data103, the CPU10calculates the attitude of the terminal device7based on these data. Here, the CPU10can know the amount of rotation per unit time (the amount of change of the attitude) of the terminal device7from the second angular velocity data102. In a state where the terminal device7is substantially stationary, the acceleration acting upon the terminal device7means the gravitational acceleration, and it is therefore possible to know, from the second acceleration data101, the direction of gravity acting upon the terminal device7, the attitude of the terminal device7with respect to the direction of gravity). It is possible to know, from the azimuthal direction data103, a predetermined azimuthal direction with respect to the terminal device7(i.e., the attitude of the terminal device7with respect to a predetermined azimuthal direction). Note that even in a case where there is a magnetic field other than the geomagnetic field, it is possible to know the amount of rotation of the terminal device7. Therefore, the CPU10can calculate the attitude of the terminal device7based on the second acceleration data101, the second angular velocity data102and the azimuthal direction data103. Note that while the attitude of the terminal device7is calculated based on the three data in the present embodiment, the attitude may be calculated based on one or two of the three data in other embodiments.

Note that while the specific method for calculating the attitude of the terminal device7may be any method, it is for example a method in which the attitude calculated based on the angular velocity represented by the second angular velocity data102is corrected using the second acceleration data101and the azimuthal direction data103. Specifically, the CPU10first calculates the attitude of the terminal device7based on the second angular velocity data102. Note that the method for calculating the attitude based on angular velocity may be similar to the method of step S23. Next, the CPU10corrects the attitude calculated based on the angular velocity by the attitude calculated based on the second acceleration data101and/or the attitude calculated based on the azimuthal direction data103at an appropriate point in time (e.g., when the terminal device7is close to being stationary). Note that the method for correcting the attitude based on the angular velocity by the attitude based on the acceleration may be similar to the method for calculating the attitude of the controller5described above. In a case where the attitude based on the angular velocity is corrected by the attitude based on the azimuthal direction data, the CPU10may bring the attitude based on the angular velocity closer to the attitude based on the azimuthal direction data at a predetermined rate. As described above, the CPU10can accurately calculate the attitude of the terminal device7.

Note that since the controller5includes the image capturing/processing section35which is infrared detection means, the game device3can obtain the marker coordinate data96. Therefore, for the controller5, the game device3can know, from the marker coordinate data96, the absolute attitude in the real space (the attitude of the controller5in the coordinate system set in the real space). On the other hand, the terminal device7does not include infrared detection means such as the image capturing/processing section35. Therefore, the game device3cannot know, only from the second acceleration data101and the second angular velocity data102, the absolute attitude in the real space for the rotation direction about the direction of gravity. In view of this, the present embodiment employs a configuration where the terminal device7includes the magnetic sensor62, and the game device3obtains the azimuthal direction data103. Then, for the rotation direction about the direction of gravity, the game device3can calculate the absolute attitude in the real space from the azimuthal direction data103, and it is possible to more accurately calculate the attitude of the terminal device7.

As a specific process of step S24, the CPU10reads out the second acceleration data101, the second angular velocity data102, and the azimuthal direction data103from the main memory, and calculates the attitude of the terminal device7based on these data. Then, the calculated data representing the attitude of the terminal device7is stored in the main memory as the terminal attitude data109. The process of step S25is performed, following step S24.

In step S25, the CPU10performs a recognition process for a camera image. That is, the CPU10performs a predetermined recognition process on the camera image data104. The recognition process may be any process as long as it detects any feature from the camera image to output the results of the detection. For example, where the face of the player is included in the camera image, it may be a process of recognizing the face. Specifically, it may be a process of detecting parts of the face (eyes, nose, mouth, etc.) or a process of detecting the expression of the face. The data representing the results of the recognition process is stored in the main memory as the image recognition data110. The process of step S26is performed, following step S25.

In step S26, the CPU10performs a recognition process for the microphone sound. That is, the CPU10performs a predetermined recognition process on the microphone sound data105. The recognition process may be any process as long as it detects any feature from the microphone sound to output the results of the detection. For example, it may be a process of detecting an instruction of the player from the microphone sound or a process of simply detecting the sound volume of the microphone sound. The data representing the results of the recognition process is stored in the main memory as the sound recognition data111. The process of step S27is performed, following step S26.

In step S27, the CPU10performs the game process in accordance with a game input. Herein, the game input may be any data as long as it is data transmitted from the controller5or the terminal device7, or data obtained from such data. Specifically, the game input may be any of various data included in the controller operation data92and the terminal operation data97, as well as data obtained from such data (the controller attitude data108, the terminal attitude data109, the image recognition data110, and the sound recognition data111). The content of the game process in step S27may be any content, and it may be, for example, a process of controlling the action of an object (character) appearing in the game, a process of controlling a virtual camera, or a process of moving a cursor displayed on the screen. It may also be a process of using the camera image (or a portion thereof) as a game image, a process of using the microphone sound as a game sound, etc. Note that examples of the game process will be described later. In step S27, data representing the results of the game control process are stored in the main memory, such as, for example, data of various parameters set for the character (object) appearing in the game, data of parameters regarding the virtual camera provided in the game space, and score data. After step S27, the CPU10ends the game control process of step S4.

Referring back toFIG. 12, in step S5, a television game image to be displayed on the television2is generated by the CPU10and the GPU11b. That is, the CPU10and the CPU11bread out data representing the results of the game control process of step S4from the main memory and readout data necessary for generating a game image from the VRAM11dto generate a game image. The game image may be any image as long as it represents the results of the game control process of step S4, and it may be generated by any method. For example, the game image generation method may be a method in which a virtual camera is provided in the virtual game space, and a three-dimensional CG image is generated by calculating the game space as seen from the virtual camera, or a method in which a two-dimensional image is generated (without using a virtual camera). The generated television game image is stored in the VRAM11d. The process of step S6is performed, following step S5.

In step S6, a terminal game image to be displayed on the terminal device7is generated by the CPU10and the GPU11b. As with the television game image, the terminal game image may be any image as long as it represents the results of the game control process of step S4, and it may be generated by any method. The terminal game image may be generated by a method similar to that for the television game image or may be generated by a different method. The generated terminal game image is stored in the VRAM11d. Note that depending on the content of the game, the television game image and the terminal game image may be the same, in which case it is not necessary to perform the process of generating a game image in step S6. The process of step S7is performed, following step S6.

In step S7, a television game sound to be outputted to the speaker2aof the television2is generated. That is, the CPU10has the DSP11cgenerate a game sound in accordance with the results of the game control process of step S4. Note that the generated game sound may be, for example, a sound effect of the game, the voice of a character appearing in the game, BCM, etc. The process of step S8is performed, following step S7.

In step S8, a terminal game sound to be outputted to the speaker67of the terminal device7is generated. That is, the CPU10has the DSP11cgenerate a game sound in accordance with the results of the game control process of step S4. Note that the terminal game sound may be the same as, or different from, the television game sound. They may be partially different from each other, e.g., differing from each other with the sound effect but being the same with the BCM. Note that in a case where the television game sound and the terminal game sound are the same, the game sound generating process may not be performed in step S8. The process of step S9is performed, following step S8.

In step S9, the CPU10outputs a game image and a game sound to the television2. Specifically, the CPU10sends the data of the television game image stored in the VRAM11dand the data of the television game sound generated by the DSP11cin step S7to the AV-IC15. In response to this, the AV-IC15outputs the image and sound data to the television2via the AV connector16. Thus, the television game image is displayed on the television2, and the television game sound is outputted from the speaker2a. The process of step S10is performed, following step S9.

In step S10, the CPU10transmits a game image and a game sound to the terminal device7. Specifically, the image data which is a terminal game image stored in the VRAM11dand the sound data generated by the DSP11cin step S8are sent by the CPU10to the codec LSI27, and are subjected to a predetermined compression process by the codec LSI27. Moreover, the image and sound data which have been subjected to the compression process are transmitted by the terminal communication module28to the terminal device7via the antenna29. The terminal device7receives the image and sound data transmitted from the game device3by the wireless module70, and the data are subjected to a predetermined expansion process by the codec LSI66. The image data which has been subjected to the expansion process is outputted to the LCD51, and the sound data which has been subjected to the expansion process is outputted to the sound IC68. Thus, the terminal game image is displayed on the LCD51, and the terminal game sound is outputted from the speaker67. The process of step S11is performed, following step S10.

In step S11, the CPU10determines whether the game should be ended. The determination of step S11is made based on, for example, whether the game has been over, the user has given an instruction to quit the game, etc. If the determination result of step S11is negative, the process of step S2is performed again. On the other hand, if the determination result of step S11is affirmative, the CPU10ends the game process shown inFIG. 12. The series of processes through steps S2to S11is repeatedly performed until it is determined in step S11that the game should be ended.

As described above, in the present embodiment, the terminal device7includes the touch panel52, and an inertia sensor such as the acceleration sensor63or the gyrosensor64, and the outputs of the touch panel52and the inertia sensor are transmitted as operation data to the game device3, and used as a game input (steps S3and S4). Moreover, the terminal device7includes a display device (the LCD51), and a game image obtained by the game process is displayed on the LCD51(steps S6and S10). Therefore, the user can perform an operation of directly touching on the game image using the touch panel52, and an operation of moving the LCD51itself on which the game image is displayed (since the movement of the terminal device7is detected by the inertia sensor). With these operations, the user can play a game with such game play as if the user were directly operating the game image, and it is therefore possible to provide a game with novel game play such as the first and second game examples to be described later, for example.

Moreover, in the present embodiment, the terminal device7includes the analog stick53and the operation button54which can be operated while holding the terminal device7, and the game device3can use, as a game input, the operation performed on the analog stick53and the operation button54(steps S3and S4). Therefore, even where the game image is directly operated as described above, the user can perform a more detailed game operation through the button operation and the stick operation.

Moreover, in the present embodiment, the terminal device7includes the camera56and the microphone69, and data of the camera image captured by the camera56and data of the microphone sound detected by the microphone69are transmitted to the game device3(step S3). Therefore, with the game device3, since the camera image and/or microphone sound can be used as a game input, the user can perform a game operation through an operation of capturing an image with the camera56or an operation of inputting sound to the microphone69. Note that since these operations can be performed while holding the terminal device7, the user can perform a greater variety of game operations by performing such operations when directly operating the game image as described above.

In the present embodiment, since a game image is displayed on the LCD51which is the terminal device7of a portable type (steps S6and S10), the user can arbitrarily place the terminal device7. Therefore, where the controller5is operated while being pointed toward the marker, the user can play a game while pointing the controller5toward an arbitrary direction by placing the terminal device7at an arbitrary position, thus improving the degree of freedom in the operation of the controller5. Since the terminal device7can be placed at an arbitrary position, it is possible to provide a more realistic game by placing the terminal device7at a position suitable for the content of the game, as in the fifth game example to be described later, for example.

According to the present embodiment, since the game device3obtains operation data, etc., from the controller5and the terminal device7(steps S2and S3), the user can use two devices of the controller5and the terminal device7as operation means. Therefore, in the game system1, a game can be played with multiple users where the devices are used by a plurality of users, or a game can be played with a single user using the two devices.

According to the present embodiment, the game device3generates two types of game images (steps S5and S6), and the game images are displayed on the television2and the terminal device7(steps S9and S10). Thus, as the two types of game images are displayed on different devices, it is possible to provide game images that are easier for the user to view, and it is possible to improve the playability of the game. For example, where a game is played by two players, a game image from a viewpoint that is easier for one user to view may be displayed on the television2while a game image from a viewpoint that is easier for the other user to view is displayed on the terminal device7, as in the third or fourth game example to be described later, in which case each player can play the game with a viewpoint that is easier for the player to view. Even if the game is played by one player, for example, if two types of game images are displayed from two different viewpoints, as in the first, second and fifth game examples to be described later, the player can more easily grasp the state of the game space, and it is therefore possible to improve the playability of the game.

Next, specific examples of games to be played on the game system1will be described. Note that game examples to be described below may not use some of the components of the device in the game system1and may not perform some of the series of processes shown inFIGS. 12 and 13. That is, the game system1does not need to include all the components described above, and the game device3may not perform some of the series of processes shown inFIGS. 12 and 13.

The first game example is a game in which an object (a shuriken, or a throwing star) is thrown in the game space by operating the terminal device7. The player can specify the direction in which a shuriken is thrown through an operation of changing the attitude of the terminal device7and an operation of drawing a line on the touch panel52.

FIG. 14is a diagram showing the screen of the television2and the terminal device7in the first game example. InFIG. 14, a game image representing the game space is displayed on the television2and the LCD51of the terminal device7. A shuriken121, a control surface122and a target123are displayed on the television2. The control surface122(and the shuriken121) are displayed on the LCD51. In the first game example, the player plays the game by throwing the shuriken121at the target123through an operation using the terminal device7.

When throwing the shuriken121, the player first changes the attitude of the control surface122provided in the virtual game space to an intended attitude by operating the attitude of the terminal device7. That is, the CPU10calculates the attitude of the terminal device7based on the outputs of the inertia sensor (the acceleration sensor63and the gyrosensor64) and the magnetic sensor62(step S24), and changes the attitude of the control surface122based on the calculated attitude (step S27). In the first game example, the attitude of the control surface122is controlled so as to be an attitude in accordance with the attitude of the terminal device7in the real space. That is, the player can change the attitude of the control surface122in the game space by changing the attitude of the terminal device7(the control surface122displayed on the terminal device7). Note that in the first game example, the position of the control surface122is fixed at a predetermined position in the game space.

Next, the player performs an operation of drawing a line on the touch panel52using a stylus124, or the like (see an arrow shown inFIG. 14). Here, in the first game example, the control surface122is displayed on the LCD51of the terminal device7so that the input surface of the touch panel52and the control surface122correspond to each other. Therefore, based on the line drawn on the touch panel52, it is possible to calculate the direction on the control surface122(the direction represented by the line). The shuriken121is thrown in a direction thus determined. As described above, the CPU10performs a process of calculating the direction on the control surface122from the touch position data100of the touch panel52, and moving the shuriken121in the calculated direction (step S27). Note that the CPU10may control the speed of the shuriken121in accordance with the length of the line or the speed at which the line is drawn, for example.

As described above, in the first game example, the game device3can move the control surface122in accordance with the movement (attitude) of the terminal device7by using the output of the inertia sensor as a game input, and identify the direction on the control surface122by using the output of the touch panel52as a game input. Thus, the player can move the game image displayed on the terminal device7(the image of the control surface122) and perform a touch operation on the game image, and can therefore play a game with such novel game play as if the player were directly operating the game image.

In the first game example, it is possible to easily specify a direction in a three-dimensional space by using sensor outputs of the inertia sensor and the touch panel52as a game input. That is, the player can easily specify a direction with such an intuitive operation as if the player were actually inputting a direction in the space, by actually adjusting the attitude of the terminal device7with one hand while inputting a direction with a line on the touch panel52with the other hand. Moreover, since the player can perform the operation on the attitude of the terminal device7and the input operation on the touch panel52simultaneously in parallel to each other, it is possible to quickly perform the operation of specifying a direction in a three-dimensional space.

In the first game example, the control surface122is displayed across the entire screen of the terminal device7so as to facilitate the touch input operation on the control surface122. On the other hand, the television2displays an image of the game space including the entire control surface122and the target123(seeFIG. 14) so that it is easy to grasp the attitude of the control surface122and aim at the target123. That is, in step S27, the first virtual camera for generating the television game image is set so that the entire control surface122and the target123are included in the range of viewing field, whereas the second virtual camera for generating the terminal game image is set so that the screen of the LCD51(the input surface of the touch panel52) and the control surface122coincide with each other on the screen. Therefore, in the first game example, images of the game space as seen from different viewpoints are displayed on the television2and on the terminal device7, thereby facilitating the game operation.

Note that the game using sensor outputs of the inertia sensor and the touch panel52as a game input is not limited to the first game example described above, and may be any of various game examples. As is the first game example, the second game example is a game in which an object (cannonball) is thrown in the game space by operating the terminal device7. The player can specify the direction in which the cannonball is thrown through an operation of changing the attitude of the terminal device7and an operation of specifying a position on the touch panel52.

FIG. 15is a diagram showing the screen of the television2and the terminal device7in the second game example. InFIG. 15, a cannon131, a cannonball132and a target133are displayed on the television2. The cannonball132and the target133are displayed on the terminal device7. The terminal game image displayed on the terminal device7is an image of the game space as seen from the position of the cannon131.

In the second game example, the player can change the range of display to be displayed on the terminal device7as the terminal game image by operating the attitude of the terminal device7. That is, the CPU10calculates the attitude of the terminal device7based on the outputs of the inertia sensors (the acceleration sensor63and the gyrosensor64) and the magnetic sensor62(step S24), and controls the position and the attitude of the second virtual camera for generating the terminal game image based on the calculated attitude (step S27). Specifically, the second virtual camera is placed at the position of the cannon131and the orientation (attitude) thereof is controlled in accordance with the attitude of the terminal device7. Thus, the player can change the range of the game space to be displayed on the terminal device7by changing the attitude of the terminal device7.

In the second game example, the player specifies the direction in which the cannonball132is to be thrown by an operation of inputting a point on the touch panel52(a touch operation). Specifically, as the process of step S27, the CPU10calculates the position (control position) in the game space corresponding to the touch position, and calculates, as the throwing direction, the direction from a predetermined position in the game space (e.g., the position of the cannon131) to the control position. Then, the CPU10performs a process of moving the cannonball132in the throwing direction. Thus, while the player performs an operation of drawing a line on the touch panel52in the first game example, the player performs an operation of specifying a point on the touch panel52in the second game example. Note that the control position can be calculated by setting a control surface similar to that of the first game example (note however that the control surface is not displayed in the second game example). That is, the position on the control surface corresponding to the touch position can be calculated as the control position by placing the control surface in accordance with the attitude of the second virtual camera so as to correspond to the display range of the terminal device7(specifically, the control surface rotates about the position of the cannon131in accordance with the change in the attitude of the terminal device7).

In the second game example, the game device3can change the display range of the terminal game image in accordance with the movement (attitude) of the terminal device7by using the output of the inertia sensor as a game input, and can specify a direction in the game space (the direction in which the cannonball132is thrown) by using the touch input specifying a position within the display range as a game input. Thus, also in the second game example, as in the first game example, the player can move the game image displayed on the terminal device7or perform a touch operation on the game image, and can therefore play a game with such novel game play as if the player were directly operating the game image.

Also in the second game example, as in the first game example, the player can easily specify a direction with such an intuitive operation as if the player were actually inputting a direction in the space, by actually adjusting the attitude of the terminal device7with one hand while performing a touch input on the touch panel52with the other hand. Moreover, since the player can perform an operation on the attitude of the terminal device7and an input operation on the touch panel52simultaneously in parallel to each other, it is possible to quickly perform the operation of specifying a direction in a three-dimensional space.

Note that in the second game example, while the image displayed on the television2may be an image from the same viewpoint as the terminal device7, the game device3displays an image from a different viewpoint inFIG. 15. That is, while the second virtual camera for generating the terminal game image is set at the position of the cannon131, the first virtual camera for generating the television game image is set at a position behind the cannon131. Here, for example, if a range that cannot be seen on the screen of the terminal device7is displayed on the television2, it is possible to realize such gameplay that the player aims at the target133, which cannot be seen on the screen of the terminal device7, while looking at the screen of the television2. Thus, by having different display ranges for the television2and for the terminal device7, it is possible not only to make it easier to grasp the state of the game space but also to further improve the playability of the game.

As described above, according to the present embodiment, since the terminal device7including the touch panel52and the inertia sensor can be used as a controller device, it is possible to realize a game with such game play as if the player were directly operating the game image, as in the first and second game examples.

Referring now toFIGS. 16 and 17, the third game example will be described. The third game example is a baseball game in which two players compete with each other. That is, the first player uses the controller5to control a batter, while the second player uses the terminal device7to control a pitcher. The television2and the terminal device7display game images which are easy for the respective players to perform operations with.

FIG. 16is a diagram showing an example of a television game image displayed on the television2in the third game example. The television game image shown inFIG. 16is an image primarily for the first player. That is, the television game image represents the game space showing a pitcher (pitcher object)142which is the object to be controlled by the second player as seen from the side of a batter (batter object)141which is the object to be controlled by the first player. The first virtual camera for generating the television game image is placed at a position behind the batter141so as to be directed from the batter141toward the pitcher142.

On the other hand,FIG. 17is a diagram showing an example of a terminal game image displayed on the terminal device7in the third game example. The terminal game image shown inFIG. 17is an image primarily for the second player. That is, the terminal game image represents the game space showing the batter141which is the object to be controlled by the first player as seen from the side of the pitcher142which is the object to be controlled by the second player. Specifically, in step S27, the CPU10controls the second virtual camera used for generating the terminal game image based on the attitude of the terminal device7. The attitude of the second virtual camera is calculated so as to correspond to the attitude of the terminal device7, as in the second game example described above. The position of the second virtual camera is fixed at a predetermined position. Note that the terminal game image includes a cursor143for indicating the direction in which the pitcher142is throwing the ball.

Note that the method by which the batter141is controlled by the first player, and the method by which the pitcher142is controlled by the second player may be any method. For example, the CPU10may detect a swing operation on the controller5based on output data of the inertia sensor of the controller5, and have the batter141swing the bat in response to the swing operation. For example, the CPU10may move the cursor143in accordance with an operation on the analog stick53, and have the pitcher142throw the ball to a position indicated by the cursor143when a predetermined one of the operation buttons54is pressed. The cursor143may be moved in accordance with the attitude of the terminal device7, instead of an operation on the analog stick53.

As described above, in the third game example, game images are generated from different viewpoints for the television2and for the terminal device7, thus providing game images that are easy to view and easy to operate with for the respective players.

In the third game example, two virtual cameras are set in a single game space so as to display two types of game images of the game space as seen from the virtual cameras (FIGS. 16 and 17). Therefore, for the two types of game images generated in the third game example, most of the game processes performed on the game space (e.g., controlling an object in the game space) are common, and the game images can be generated simply by performing the drawing process twice on a common game space, thus providing an advantage that the process efficiency is higher than when the game processes are performed separately.

In the third game example, since the cursor143representing the pitching direction is displayed only on the side of the terminal device7, the first player cannot see the position indicated by the cursor143. Therefore, the game does not have such a problem that the first player gets to know the pitching direction to the disadvantage of the second player. Thus, in the present embodiment, if there is a problem in the game for one player if the other player sees a game image, the game image can be displayed on the terminal device7. Thus, it is possible to prevent a problem of, for example, detracting from the strategic aspect of the game. Note that in other embodiments, the game device3may display the terminal game image on the television2along with the television game image depending on the content of the game (e.g., where no such problem as described above occurs even if the terminal game image is seen by the first player).

Referring now toFIGS. 18 and 19, the fourth game example will be described. The fourth game example is a shooting game of such a format where two players cooperate with each other. That is, the first player uses the controller5to perform an operation of moving an airplane, and the second player uses the terminal device7to perform an operation of controlling the cannon-firing direction of the airplane. In the fourth game example, as in the third game example, game images that are easy for the respective players to perform game operations with are displayed on the television2and on the terminal device7.

FIG. 18is a diagram showing an example of a television game image displayed on the television2in the fourth game example.FIG. 19is a diagram showing an example of a terminal game image displayed on the terminal device7in the fourth game example. As shown inFIG. 18, an airplane (airplane object)151and a target (balloon object)153appear in the virtual game space in the fourth game example. The airplane151has a cannon (cannon object)152.

As shown inFIG. 18, an image of the game space including the airplane151is displayed as the television game image. The first virtual camera for generating the television game image is set so as to produce an image of the game space showing the airplane151as seen from behind. That is, the first virtual camera is placed behind the airplane151at such an attitude that the airplane151is included in the image-capturing range (range of viewing field). The first virtual camera is controlled so as to be moved in accordance with the movement of the airplane151. That is, in the process of step S27, the CPU10controls the movement of the airplane151based on the controller operation data, and also controls the position and the attitude of the first virtual camera. Thus, the position and the attitude of the first virtual camera are controlled in accordance with the operation of the first player.

On the other hand, as shown inFIG. 19, an image of the game space as seen from the airplane151(more specifically, the cannon152) is displayed as the terminal game image. Therefore, the second virtual camera for generating the terminal game image is placed at the position of the airplane151(more specifically, the position of the cannon152). In the process of step S27, based on the controller operation data, the CPU10controls the movement of the airplane151and also controls the position of the second virtual camera. Note that the second virtual camera may be placed at a position around the airplane151or the cannon152(e.g., a position slightly behind the cannon152). As described above, the position of the second virtual camera is controlled by the operation of the first player (operating the movement of the airplane151). Therefore, in the fourth game example, the first virtual camera and the second virtual camera move in cooperation with each other.

An image of the game space as seen in the firing direction of the cannon152is displayed as the terminal game image. Here, the firing direction of the cannon152is controlled so as to correspond to the attitude of the terminal device7. That is, in the present embodiment, the attitude of the second virtual camera is controlled so that the line-of-sight direction of the second virtual camera coincides with the firing direction of the cannon152. In the process of step S27, the CPU10controls the orientation of the cannon152and the attitude of the second virtual camera in accordance with the attitude of the terminal device7calculated in step S24. Thus, the attitude of the second virtual camera is controlled by the operation of the second player. The second player can change the firing direction of the cannon152by changing the attitude of the terminal device7.

Note that when firing a cannonball from the cannon152, the second player presses a predetermined button of the terminal device7. When the predetermined button is pressed, a cannonball is fired in the orientation of the cannon152. In the terminal game image, a sight154is displayed at the center of the screen of the LCD51, and the cannonball is fired in the direction indicated by the sight154.

As described above, in the fourth game example, the first player operates the airplane151(so that it moves in the direction of an intended target153, for example) while looking primarily at the television game image (FIG. 18) representing the game space viewing in the traveling direction of the airplane151. On the other hand, the second player operates the cannon152while looking primarily at the terminal game image (FIG. 19) representing the game space viewing in the firing direction of the cannon152. Thus, in the fourth game example, in a game of such a format where two players cooperate with each other, game images that are easy to view and easy to operate with for the respective players are displayed on the television2and on the terminal device7.

In the fourth game example, the positions of the first virtual camera and the second virtual camera are controlled by the operation of the first player, and the attitude of the second virtual camera is controlled by the operation of the second player. That is, in the present embodiment, the position or the attitude of a virtual camera changes in accordance with the game operation by each player, thereby changing the display range of the game space to be displayed on each display device. Since the display range of the game space to be displayed on the display device changes in accordance with the operation of each player, each player can realize that one's game operation is sufficiently reflected in the progress of the game, and can thus enjoy the game sufficiently.

Note that in the fourth game example, a game image as seen from behind the airplane151is displayed on the television2, and a game image as seen from the position of the cannon of the airplane151is displayed on the terminal device7. Here, in other game examples, the game device3may display a game image as seen from behind the airplane151on the terminal device7, and a game image as seen from the position of the cannon152of the airplane151on the television2. Then, the roles of the players are switched around from the fourth game example so that the first player uses the controller5to operate the cannon152while the second player uses the terminal device7to operate the airplane151.

Referring now toFIG. 20, the fifth game example will be described. The fifth game example is a game in which a player uses the controller5to perform an operation, and the terminal device7is used as a display device, not as a controller device. Specifically, the fifth game example is a golf game, wherein the game device3has a player character in the virtual game space take a golf swing in accordance with the player performing an operation (swing operation) of swinging the controller5as if it were a golf club.

FIG. 20is a diagram showing how the game system1is used in the fifth game example. InFIG. 20, an image of the game space including (an object of) a player character161and (an object of) a golf club162is displayed on the screen of the television2. Note that (an object of) a ball163placed in the game space is also displayed on the television2though it is not shown inFIG. 20as being hidden behind the golf club162. On the other hand, as shown inFIG. 20, the terminal device7is placed on the floor surface in front of the television2so that the screen of the LCD51is facing vertically upward. An image representing the ball163, an image representing a part of the golf club162(specifically, a head162aof the golf club), and an image representing the ground of the game space are displayed on the terminal device7. The terminal game image is an image of the vicinity of the ball as seen from above.

When playing the game, a player160stands near the terminal device7, and performs a swing operation of swinging the controller5as if it were a golf club. Then, in step S27, the CPU10controls the position and the attitude of the golf club162in the game space in accordance with the attitude of the controller5calculated in the process of step S23. Specifically, the golf club162is controlled so that the golf club162in the game space hits the ball163when the tip direction of the controller5(the Z-axis positive direction shown inFIG. 3) is pointing toward the image of the ball163displayed on the LCD51.

When the tip direction of the controller5is pointing toward the LCD51, an image (head image)164representing a part of the golf club162is displayed on the LCD51(seeFIG. 20). Note that for the terminal game image, the image of the ball163may be shown in the actual size, and the orientation of the head image164may be shown to rotate in accordance with the rotation of the controller5about the Z axis, in order to enhance the reality. The terminal game image may be generated using a virtual camera provided in the game space, or generated using pre-prepared image data. When it is generated using pre-prepared image data, detailed and realistic images can be generated with low computational load without constructing the terrain model of a golf course in detail.

As a result of the player160performing the swing operation so as to swing the golf club162, if the golf club162hits the ball163, the ball163travels (flies). That is, the CPU10determines in step S27whether the golf club162and the ball163have contacted each other, and moves the ball163when there has been a contact. Here, the television game image is generated so that the ball163after the travel is included therein. That is, the CPU10controls the position and the attitude of the first virtual camera for generating the television game image so that the traveling ball is included in the image-capturing range thereof. On the other hand, on the terminal device7, when the golf club162hits the ball163, the image of the ball163is moved and immediately disappears to the outside of the screen. Thus, in the fifth game example, the travel of the ball is displayed primarily on the television2, and the player160can check, on the television game image, the destination of the ball hit by the swing operation.

As described above, in the fifth game example, the player160can swing the golf club162by swinging the controller5(have the player character161swing the golf club162). Here, in the fifth game example, the golf club162in the game space is controlled to hit the ball163when the tip direction of the controller5is pointing toward the image of the ball163displayed on the LCD51. Therefore, the player can perform the swing operation and thereby feel as if the player were taking a swing with an actual golf club, thus making the swing operation feel more realistic.

Moreover, in the fifth game example, the head image164is displayed on the LCD51when the tip direction of the controller5is pointing toward the terminal device7. Therefore, as the player points the tip direction of the controller5toward the terminal device7, the player can feel that the attitude of the golf club162in the virtual space corresponds to the attitude of the controller5in the real space, thus making the swing operation feel more realistic.

As described above, in the fifth game example, where the terminal device7is used as a display device, it is possible to make the operation using the controller5feel more realistic by locating the terminal device7at an appropriate position.

In the fifth game example, the terminal device7is placed on the floor surface, and an image representing the game space showing only the vicinity of the ball163is displayed on the terminal device7. Therefore, the position/attitude of the entire golf club162in the game space cannot be displayed on the terminal device7, and how the ball163travels after the swing operation cannot be displayed on the terminal device7. In view of this, in the fifth game example, the entire golf club162is displayed on the television2before the ball163travels, and how the ball163travels is displayed on the television2after the ball163starts traveling. Thus, in the fifth game example, it is possible to provide the player with a realistic operation, and game images that are easy to view can be presented to the player by using two screens of the television2and the terminal device7.

In the fifth game example, the marker section55of the terminal device7is used for calculating the attitude of the controller5. That is, the CPU10lights the marker section55(does not light the marker device6) in the initialization process of step S1, and the CPU10calculates the attitude of the controller5based on the marker coordinate data96in step S23. Then, it is possible to accurately determine whether the tip direction of the controller5is in an attitude pointing toward the marker section55. Note that while steps S21and S22do not have to be performed in the fifth game example, a marker or markers to be lit may be changed in the middle of the game in other game examples by performing the process of steps S21and S22. For example, the CPU10may determined in step S21whether the tip direction of the controller5is pointing in the direction of gravity based on the first acceleration data94, and in step S22, the CPU10may light the marker section55if it is pointing in the direction of gravity and light the marker device6if it is not pointing in the direction of gravity. Then, where the tip direction of the controller5is pointing in the direction of gravity, the attitude of the controller5can be calculated with high precision by obtaining marker coordinate data of the marker section55, and where the tip direction of the controller5is pointing toward the television2, the attitude of the controller5can be calculated with high precision by obtaining marker coordinate data of the marker device6.

As described above in the fifth game example, in the game system1, the terminal device7canoe placed at an arbitrary position and used as a display device. Then, where the marker coordinate data is used as a game input, the controller5can be used while pointing in an arbitrary direction by setting the terminal device7at an intended position, in addition to using the controller5while pointing toward the television2. That is, according to the present embodiment, since the orientation in which the controller5is used is not limited to any particular orientation, it is possible to improve the degree of freedom in operations to be performed on the controller5.

[7. Other Operation Examples of Game System]

In the game system1, it is possible to perform operations for playing various games as described above. While the terminal device7can be used as a portable display or a second display, it may also be used as a controller for making a touch input or a motion-based input, and it is therefore possible to realize a wide variety of games with the game system1. Operations as follows can also be performed, including applications other than games.

(Operation Example where Player Plays Game Only Using Terminal Device7)

In the present embodiment, the terminal device7can function as a display device and can also function as a controller device. Therefore, one can use the terminal device7like a portable game device by using the terminal device7as display means and as operation means and without using the television2and the controller5.

Specifically, according to the game process shown inFIG. 12, the CPU10obtains the terminal operation data97from the terminal device7in step S3, and performs a game process using only the terminal operation data97as a game input (without using the controller operation data) in step S4. Then, a game image is generated in step S6, and the game image is transmitted to the terminal device7in step S10. Note that steps S2, S5and S9may not be performed. Thus, a game process is performed in accordance with an operation on the terminal device7, and a game image representing the game process results is displayed on the terminal device7. Then, the terminal device7can be used as a portable game device (though the game process is actually performed by the game device). Therefore, according to the present embodiment, the user can play a game using the terminal device7even in a case where a game image cannot be displayed on the television2for reasons such as the television2being used (e.g., someone else watching a TV broadcast).

Note that in addition to the game image, the CPU10may transmit an image of the menu screen described above to be displayed after power-up to the terminal device7so that the image is displayed thereon. This is convenient because the player can play a game without using the television2from the beginning.

Moreover, in the above description, the display device on which the game image is displayed can be changed from the terminal device7to the television2in the middle of the game. Specifically, the CPU10can further perform step S9to output the game image to the television2. Note that the image to be outputted to the television2in step S9is the same as the game image to be transmitted to the terminal device7in step S10. Then, by switching the input of the television2so that the input from the game device3is displayed thereon, the same game image as that on the terminal device7is displayed on the television2. Thus, the display device on which the game image is displayed can be changed to the television2. Note that after the game image is displayed on the television2, the display of the screen of the terminal device7may be turned OFF.

Note that the game system1may be such that the infrared remote controller signal for the television2can be outputted from infrared outputting means (the marker device6, the marker section55or the infrared communication module72). Then, the game device3can perform an operation on the television2by outputting the infrared remote controller signal from the infrared outputting means in accordance with an operation on the terminal device7. In such a case, since the user can operate the television2by using the terminal device7without operating the remote controller of the television2, it is convenient when, for example, switching the input of the television2from one to another as described above.

(Operation Example where System Communicates with Another Device Via Network)

Since the game device3has a network connection function as described above, the game system1can be used in a case where it communicates with an external device via a network.FIG. 21is a diagram showing how devices included in the game system1are connected with one another in a case where the game system1is connected to an external device via a network. As shown inFIG. 21, the game device3can communicate with an external device201via a network200.

Where the external device201and the game device3can communicate with each other as described above, the game system1can communicate with the external device201using the terminal device7as an interface. For example, the game system1can be used, as a video telephone by exchanging image and sound between the external device201and the terminal device7. Specifically, the game device3receives the image and sound from the external device201(the image and the sound of the other person) via the network200, and transmits the received image and sound to the terminal device7. Then, the terminal device7displays the image from the external device201on the LCD51and outputs from the speaker67the sound from the external device201. The game device3receives from the terminal device7the camera image captured by the camera56and the microphone sound detected by the microphone69, and transmits the camera image and the microphone sound to the external device201via the network200. The game system1can be used as a video telephone as the game device3repeats the exchange of the image and the sound described above with the external device201.

Note that since the terminal device7is portable in the present embodiment, the user can use the terminal device7at an arbitrary position or direct the camera56in an arbitrary direction. In the present embodiment, since the terminal device7includes the touch panel52, the game device3can transmit the input information made on the touch panel52(the touch position data100) to the external device201. For example, the game system1can be used as a so-called e-learning system when outputting from the terminal device7the image and sound from the external device201, and transmitting characters, etc., the user has written on the touch panel52to the external device201.

(Operation Example where System Cooperates with TV Broadcasting)

The game system1can also operate in cooperation with TV broadcasting when a TV broadcast is being watched on the television2. That is, when a TV program is being watched on the television2, the game system1can output on the terminal device7information regarding the TV program, etc. An operation example where the game system1operates in cooperation with TV broadcasting will now be described.

In the operation example described above, the game device3can communicate with a server via a network (in other words, the external device201shown inFIG. 21is the server). The server stores, for each channel of TV broadcasting, various information relating to TV broadcasting (TV information). The TV information may be program-related information such as subtitles and cast information, EPG (Electronic Program Guide) information, or information to be broadcast as a data broadcast. The TV information may be image, sound, text, or information of a combination thereof. The number of servers does not need to be one, a server may be provided for each channel or each program of TV broadcasting, and the game device3may be able to communicate with the servers.

Where video/sound of a TV broadcast is being outputted from the television2, the game device3prompts the user to input the channel of the TV broadcast being watched by using the terminal device7. Then, a request is given via the network to the server to transmit TV information corresponding to the inputted channel. In response to this, the server transmits data of TV information corresponding to the channel. When receiving data transmitted from the server, the game device3outputs the received data to the terminal device7. The terminal device7displays image and text data of that data on the LCD51, and outputs sound data from the speaker. As described above, the user can enjoy information relating to the TV program being watched currently, etc., using the terminal device7.

As described above, the game system1can communicate with an external device (server) via a network so that information linked to TV broadcast ing can be presented to the user by the terminal device7. Particularly, this gives great convenience since the terminal device7is portable in the present embodiment, and the user can use the terminal device7at an arbitrary position.

As described above, in the present embodiment, the user can use the terminal device7in various applications/forms, in addition to game applications.

The above embodiment is an example of how the present invention can be carried out, and the present invention may also be carried out with, for example, the following configurations in other embodiments.

(Variation Using Plurality of Terminal Devices)

While the game system1includes only one terminal device in the above embodiment, the game system1may include a plurality of terminal devices. That is, the game device3may be able to wirelessly communicate with each of a plurality of terminal devices, wherein the game device3transmits game image data, game sound data and control data to each terminal device, and receives operation data, camera image data and microphone sound data from each terminal device. Note that while the game device3wirelessly communicates with the plurality of terminal devices, the game device3can realize the wireless communication with the terminal devices by time division multiple access or frequency division multiple access.

In a case where there are a plurality of terminal devices as described above, a greater variety of games can be played using the game system. For example, where the game system1includes two terminal devices, the game system1has three display devices, and the game system1can therefore generate game images for the three players and display the game images on the respective display devices. Where the game system1includes two terminal devices, two players can simultaneously play a game in which a controller and a terminal device are used as a set (e.g., the fifth game example). Moreover, where the game process of step S27is performed based on marker coordinate data outputted from two controllers, two players can each perform a game operation while pointing the controller toward the marker (the marker device6or the marker section55). That is, one player can perform a game operation while pointing the controller toward the marker device6, and the other player can perform a game operation while pointing the controller toward the marker section55.

(Variation Regarding Function of Terminal Device)

In the above embodiment, the terminal device7functions as a so-called thin client terminal, and does not perform the game process. Here, in other embodiments, some of a series of game processes performed by the game device3in the above embodiment may be performed by other devices such as the terminal device7. For example, some processes (e.g., the process of generating the terminal game image) may be performed by the terminal device7. For example, in a game system including a plurality of information processing devices (game devices) that can communicate with each other, the game processes may be divided among the plurality of information processing devices.

Industrial Applicability

As described above, the present invention is applicable to, for example, a game system, a terminal device used in a game system, etc., aiming at, for example, making the player perform a novel game operation.

DESCRIPTION OF THE REFERENCE NUMERALS

1: Game system

3: Game device

4: Optical disc

6: Marker device

7: Terminal device

11e: Internal main memory

12: External main memory

19: Controller communication module

28: Terminal communication module

44: Wireless module

52: Touch panel

54: Operation button

55: Marker section

70: Wireless module

92: Controller operation data

97: Terminal operation data

98: Camera image data

105: Microphone sound data