Information processing apparatus, information processing method, and medium

An information processing apparatus, an information processing method and a medium which allow accurate control of an image pickup timing based on a result of image recognition with a simple construction. An image pickup section outputs picked up image data to an image recognition section and a delaying section. The image recognition section performs image recognition for the image data inputted thereto and, when it detects predetermined information from among the image data, outputs a trigger signal to an image fetching section to fetch an image. The delaying section delays the image data inputted thereto from the image recognition section to compensate for a delay corresponding to a time required for the image recognition by the image recognition section and outputs resulting image data to the image fetching section. The image fetching section fetches the image data inputted thereto from the delaying section in synchronism with the trigger signal inputted thereto from the image recognition section and outputs the fetched image data to the image storage section. The image storage section stores the image data outputted from the image fetching section.

BACKGROUND OF THE INVENTION

This invention relates to an information processing apparatus, an information processing method and a medium wherein an image pickup timing is controlled based on a result of image recognition.

Conventionally, an image pickup timing is inputted to a camera using various methods. According to a method, a user physically presses a shutter button. According to another method, an output signal of, for example, a sensor which detects sound, light or the like is inputted as a trigger signal to a camera.

In the method which employs a sensor for detecting sound, light or the like described above, a disturbance such as noise or scattered light sometimes hinders the sensor from outputting an accurate trigger signal, resulting in malfunction of the camera to which the trigger signal is inputted. Further, the sensor for detecting sound, light or the like must be provided in addition to an image pickup system of the camera. This complicates the construction of the camera.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide an information processing apparatus, an information processing method and a medium which allow accurate control of an image pickup timing with a simple construction and an image pickup system can be also used as a sensor.

In order to attain the object described above, according to an aspect of the present invention, there is provided an information processing apparatus, comprising image pickup means for picking up an image of an image pickup object to produce image data, detection means for detecting a variation of the state of the image pickup object from within the image data produced by the image pickup means and generating a trigger signal, and storage means for storing the image data produced by the image pickup means in synchronism with the trigger signal generated by the detection means.

In the information processing apparatus, a variation of the state of an image pickup object is detected based on image data produced by the image pickup means, and a trigger signal is generated in response to the detection. Then, the image data are stored into the storage means in synchronism with the trigger signal. Consequently, the image pickup timing can be-controlled accurately. Further, such accurate control of the image pickup timing can be realized with a simplified construction because the image pickup means is used also as a sensor for detecting a variation of the state of the image pickup object.

According to another aspect of the present invention, there is provided an information processing method, comprising a first pixel value calculation processing step of calculating a sum total of pixel values of all pixels of image data for one frame fetched prior by a predetermined interval of time by image pickup means which picks up an image of an image pickup object, a second pixel value calculation processing step of calculating a sum total of pixel values of all of the pixels of image data for one frame fetched at a present point of time by the image pickup means, a detection processing step of calculating a difference value between the value calculated in the first pixel value calculation processing step and the value calculated in the second pixel value calculation processing step and, when the difference value is greater than a reference value set in advance, determining that a variation of the state has occurred with the image pickup object and generating a trigger signal, and a storage processing step of storing the image data for one frame fetched prior by the predetermined interval of time at a point of time when the trigger signal is generated in the detection processing step.

According to a further aspect of the present invention, there is provided a medium which causes a computer to execute an information processing program comprising a first pixel value calculation processing step of calculating a sum total of pixel values of all pixels of image data for one frame fetched prior by a predetermined interval of time by image pickup means which picks up an image of an image pickup object, a second pixel value calculation processing step of calculating a sum total of pixel values of all of the pixels of image data for one frame fetched at a present point of time by the image pickup means, a detection processing step of calculating a difference value between the value calculated in the first pixel value calculation processing step and the value calculated in the second pixel value calculation processing step and, when the difference value is greater than a reference value set in advance, determining that a variation of the state has occurred with the image pickup object and generating a trigger signal, and a storage processing step of storing the image data for one frame fetched prior by the predetermined interval of time at a point of time when the trigger signal is generated in the detection processing step.

In the information processing method and the medium, a sum total of pixel values of all pixels of image data for one frame fetched prior by a predetermined interval of time by the image pickup means which picks up an image of an image pickup object is calculated, and a sum total of pixel values of all of the pixels of image data for one frame fetched at a present point of time by the image pickup means is calculated. Then, a difference value between the values calculated in this manner is calculated. When the difference value is greater than the reference value set in advance, it is determined that a variation of the state has occurred with the image pickup object, and a trigger signal is generated. At the point of time when the trigger signal is generated, the image data for one frame fetched prior by the predetermined interval of time are stored. Consequently, the image pickup timing can be controlled accurately. Further, such accurate control of the image pickup timing can be realized with a simplified construction because the image pickup means is used also as a sensor for detecting a variation of the state of the image pickup object.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring first toFIGS. 1 to 6, there is shown a portable personal computer to which the present invention is applied. The personal computer generally denoted at1is a personal computer of the mini notebook type and includes, as basic components thereof, a body2and a display part3mounted for pivotal opening and closing movement with respect to the body2along an axis extending along a direction of the rear side of the body2.FIG. 1is a perspective view showing a personal computer1according to the present invention, the display part3is opened with respect to the body2; FIG.2is a plan view of the personal computer shown inFIG. 1;FIG. 3is a left-hand side elevational view showing the personal computer1with the display part3closed with respect to the body2;FIG. 4is a right-hand side elevational view showing the personal computer1with the display part3opened by 180 degrees with respect to the body2;FIG. 5is a front elevational view of the personal computer1shown inFIG. 3andFIG. 6is a bottom plan view of the personal computer1shown inFIG. 4.

A keyboard4and a stick type pointing device5are mounted on an upper face of the body2. The keyboard4is operated to input various characters, symbols and so forth, and the stick type pointing device5is operated to move a mouse cursor and so forth. Also a speaker8which outputs sound and a shutter button10are provided on the upper face of the body2. The shutter button10is operated to pick up an image of an object by means of a CCD video camera23which is provided on the display part3.

A pawl13is provided at an upper end of the display part3inFIG. 1, and a hole6into which the pawl13is to be fitted is provided open at a position of the body2which opposes the pawl13when the display part3is closed with respect to the body2as shown inFIG. 3. A slide lever7is mounted on a front face of the body2for sliding movement in parallel to the front face of the body2to and from a position in which it engages with the pawl13fitted in the hole6to lock the pawl13and hence lock the display part3to its closed position. When the pawl13is unlocked from the slide lever7, the display part3can be pivoted with respect to the body2. A microphone24is mounted adjacent to the pawl13. The microphone24can collect sound also from the back of the personal computer1as shown inFIG. 6.

Also a programmable power key (PPK)9is provided in the front face of the body2. An air outlet11is formed on a right-hand side wall of the body2as shown inFIG. 4, and an air inlet14is formed at a lower portion of the front face of the body2as shown inFIG. 5. A slot12for receiving a PCMCIA (Personal Computer Memory Card International Association) card (PC card) is formed on the right side of the air outlet11.

An LCD (Liquid Crystal Display)21for displaying an image thereon is provided on the front face of the display part3, and an image pickup part22is mounted at an upper end of the LCD21inFIG. 1for pivotal motion with respect to the display part3. In particular, the image pickup part22is mounted for turning movement to any position within a range of180degrees from the direction of the LCD21to the opposite direction. The CCD video camera23is mounted on the image pickup part22.

A power supply lamp PL, a battery lamp BL, a message lamp ML and other necessary lamps which may be formed from light emitting diodes (LEDs) are provided at lower portions of the display part3inFIG. 1adjacent to the body2. It is to be noted that reference numeral40shown inFIG. 3denotes a power supply switch provided on the left side face of the body2, and25shown inFIG. 5denotes an adjustment ring for adjusting the focus of the CCD video camera23. Further, reference numeral26shown inFIG. 6denotes a lid which covers over an opening through which an add-on memory is to be loaded into the body2, and reference numeral41denotes a small hole into which a pin for unlocking the locking pawl of the lid26is to be inserted.

FIG. 7shows an internal structure of the personal computer1. Referring toFIG. 7, a CPU (Central Processing Unit)52, a PC card53which is loaded into the personal computer1when necessary, a RAM (Random Access Memory)54and a graphics chip81are connected to an internal bus51. The internal bus51is connected to an external bus55, and a hard disk drive (HDD)56, an I/O (input/output) controller57, a keyboard controller58, a stick type pointing device controller59, a sound chip60, an LCD controller83, a modem50and other required devices are connected to the external bus55.

The CPU52serves as a controller for controlling various functions. The PC card53is suitably loaded in order to add an optional function to the personal computer1.

The RAM54has, at a point of time when start-up is completed, an electronic mail program (application program)54A, an auto pilot program (application program)54B and an OS (operating system; basic program)54C stored therein. The programs mentioned are transferred from the HDD56to the RAM54upon start-up.

The electronic mail program54A is a program for sending or receiving a communication message through a network to or from a communication line such as a telephone line. The electronic mail program54A has an in-coming mail capturing function as a particular function. The in-coming mail capturing function executes processing of checking a mail box93A of a mail server93for a mail addressed to that user and, if such a mail is found, capturing the same.

The auto pilot program54B is a program for successively starting a plurality of processes (or programs) set in advance in a predetermined order.

The OS (basic program software)54C controls basic operation of the computer represented by the Windows98 (trademark).

Meanwhile, the hard disk drive (HDD)56of the external bus55side has an electronic mail program56A, an auto pilot program56B and an OS (basic program software)56C stored therein. The OS56C, auto pilot program56B and electronic mail program56A in the hard disk drive56are successively transferred to and stored into the RAM54in a procedure of start-up (boot-up) processing.

The I/O controller57includes a microcontroller61provided with an I/O interface62. The microcontroller61includes, in addition to the I/O interface62, a CPU63, a RAM64and a ROM69which are connected to each other. The RAM64includes a key-input status register65, an LED (light emitting diode) control register66, a setting time register67, and a register68. The setting time register67is used to start operation of a start sequence controller76when a time (start-up condition) set in advance by the user comes. The register68stores a correspondence between a combination (start-up condition) of operation keys set in advance and an application program to be started up. When the stored combination of operation keys is inputted by the user, the stored application program (for example, an electronic mail) is started up.

The key-input status register65stores an operation key flag when the programmable power key (PPK)9for single-touch operation is pressed. The LED control register66controls the turn-on/off of the message lamp ML which indicates a start-up condition of an application program (electronic mail program) stored in the register68. The setting time register67accepts setting of an arbitrary time.

A backup battery74is connected to the microcontroller61so that stored values of the registers65,66and67may be maintained even when the power supply to the body2is off.

The ROM69in the microcontroller61has a wakeup program70, a key-input monitor program71and an LED control program72stored in advance therein. The ROM69is formed from, for example, an EEPROM (electrically erasable and programmable read only memory). The EEPROM is also called flash memory. Further, an RTC (Real-Time Clock)75which normally counts the present time is connected to the microcontroller61.

The wakeup program70in the ROM69is a program for checking based on the present time data supplied thereto from the RTC75whether or not a time set in advance in the setting time register67comes and starts up a predetermined process (or program) or they like when the set time comes. The key-input monitor program71is a program for normally supervising whether or not the PPK9is pressed by a user. The LED control program72is a program for controlling the turn-on/off of the message lamp ML.

The ROM69further has a BIOS (Basic Input/Output System)73written therein. The BIOS is a basic input/output system and is a software program for controlling the transfer (input/output) of data between an OS or application software and a peripheral equipment such as display unit, a keyboard, or a hard disk drive.

The keyboard controller58connected to the external bus55controls inputting from the keyboard4. The stick type pointing device controller59controls inputting of the stick type pointing device5.

The sound chip60fetches an input from the microphone24or supplies an audio signal to the speaker8.

The modem50can be connected to a communication network such as the Internet92or the mail server93through a public telephone line90or an Internet service provider91.

The graphics chip81connected to the internal bus51receives, as input data thereto, image data picked up by the CCD video camera23and processed in a processing block82. The graphics chip81stores video data inputted thereto from the CCD video camera23through the processing block82into a VRAM (video RAM)81A built therein and reads out and outputs the data suitably to the LCD controller83. The LCD controller83outputs the image data supplied thereto from the graphics chip81to the LCD21so that they may be displayed on the LCD21. A backlight84illuminates the LCD21from behind.

The power supply switch40is operated to switch the power supply on or off. A half press switch85is switched on when the shutter button10is pressed into a half-pressed condition. A full press switch86is switched on when the shutter button10is pressed into a full-pressed condition. A reversal switch87is switched on when the image pickup part22is turned by 180 degrees (when the CCD video camera23is turned to a direction to pick up an image on the opposite side to the LCD21).

In the personal computer1described above, image data fetched by the CCD video camera23are stored into the VRAM81A built in the graphics chip81. The image data stored in the VRAM81A are captured (stored into the hard disk56) at the timing when the shutter button10is fully pressed as a hardware switch by a user (when the full press switch86is turned on) or when, as a software switch, predetermined information is detected from within the image data stored in the VRAM81A by an image recognition application which is operating in the personal computer1.

FIG. 10shows a particular example of a display of an image pickup application program as the image recognition application mentioned above. The following description is given of SMART CAPTURE (trademark) as an example that works as the image pickup application program for capturing an image through the CCD video camera23.

When SMART CAPTURE is operating, the CPU52instructs the graphics chip81to execute the processing for displaying the image picked up by the CCD video camera23on the LCD21. Namely, the image data picked up by the CCD video camera23are processed by the processing block82in a predetermined manner and supplied to the graphics chip81through a ZV port. The graphics chip81temporarily stores the received image data into the VRAM81A and reads and outputs the stored image data to the LCD controller83. The LCD controller83outputs the received image data to the LCD21(a finder screen241) so as to be displayed on the same. Consequently, the image picked up by the CCD video camera as shown inFIG. 10, for example, is displayed as a motion picture in a window231of SMART CAPTURE.

As shown inFIG. 10, the finder screen241is arranged in the window231. The image picked up by the CCD video camera23is displayed in the finder screen241. A status display section242displays the status information of the image displayed in the finder screen241. For example, in the still picture pickup mode, a picture quality mode (FINE), a picture size (320 ×240), and a hard disk free space (Remain 1.51 GB) are displayed. In the motion picture pickup mode, a maximum recording time length, a recording time at present, and a picture size are displayed.

An option button243is operated to designate an image pickup mode or a like mode. An effect button244is operated to impart an effect to the picked up image. A capture button245has a function equivalent to that of the mechanical shutter button10(refer toFIG. 1) and operated to pick up an image. The word on the capture button245is “STILL” for still picture pickup and “VIDEO” for motion picture pickup.

An application selector menu246displays the name of an application to be used to process the picked up image. In the state shown inFIG. 10, Still Viewer (trademark), which is an application program for still image reproduction, is selected, and SMART CAPTURE is in the state for supplying the obtained image data to Still Viewer. Therefore,“Still Viewer” is displayed on the application selector menu246. A button247to the right of the application selector menu246is operated to switch between applications.

FIG. 8illustrates a functional construction of the image recognition application. Referring toFIG. 8, an image pickup section101(which corresponds to the CCD video camera23ofFIG. 7) picks up an image and outputs resulting image data to an image recognition section102and a delaying section103.

The image recognition section102performs image recognition for the image data inputted thereto from the image pickup section101. If the image recognition section102detects desired information from within the image data, then it outputs a trigger signal to an image fetching section104to instruct the image fetching section104to fetch an image. The image recognition section102detects, as the desired information mentioned above, for example, a motion vector of an object such as a person who moves across the screen, an intensity variation of light such as that when a lamp is turned on or off, or a particular image pattern such as the face of a person. The information to be detected by the image recognition section102is set by a user in advance.

The delaying section103delays the image data inputted thereto from the image pickup section101to compensate for a delay of time required for the image recognition by the image recognition section102and outputs the delayed image data to the image fetching section104. The image fetching section104fetches the image data inputted thereto from the delaying section103in response to the trigger signal inputted thereto from the image recognition section102and outputs the fetched image data to an image storage section105(which corresponds to the hard disk56ofFIG. 7). The image storage section105stores the image data outputted from the image fetching section104.

Now, image pickup processing of the image recognition application having the construction described above is described with reference toFIG. 9in connection with an example wherein the image recognition section102is set so as to detect an intensity variation of light. When a user wants, for example, to record image data of a certain room picked up when a lamp in the room is turned on or off, the user will set the image recognition section102so that it detects an intensity variation of light. In step S1, the image pickup section101starts image pickup of the room and outputs the picked up image data to the image recognition section102and the delaying section103. Accordingly, real time image data obtained by picking up an image of the room are successively inputted to the image recognition section102and the delaying section103.

In step S2, the image recognition section102extracts a luminous intensity (brightness value) of a pixel at a predetermined position for each predetermined number of frames (for example, for each 10 frames) from among the image data inputted thereto from the image pickup section101, and calculates a difference of the brightness value from another brightness value of the same pixel in another frame prior by 10 frames in time.

In step S3, the image recognition section102determines whether or not the difference in brightness value calculated in step S2is greater than a predetermined value (whether the lamp has been turned on or off). The processes in steps S2and S3are repeated until it is determined in step S3that the difference in brightness value is greater than the predetermined value. If it is determined in step S3that the difference in brightness value calculated in step S2is greater than the predetermined value, then the control advances to step S4.

In the meantime, the delaying section103delays the image data inputted thereto by a time equal to the time required for the processes of the image recognition section102in steps S2and S3and outputs the delayed image data to the image fetching section104. The delaying section103adjusts the delay time in response to the information detected by the image recognition section102.

In step S4, the image recognition section102outputs a trigger signal to the image fetching section104. The image fetching section104fetches the image data inputted thereto from the delaying section103in synchronism with the trigger signal inputted thereto from the image recognition section102, and outputs the still image data to the image storage section105. In step S5, the image storage section105stores the still image data inputted thereto from the image fetching section104.

It is to be noted that the delaying section103adjusts its delay type also in response to the type of information detected by the image recognition section102. For example, when the image recognition section102detects an intensity of light, a trigger signal is outputted when the lamp is turned on from an off state. However, it is meaningless if image data of the dark room after the lamp is turned off are recorded. Therefore, the delay time is adjusted so that image data immediately before the lamp is turned off may be recorded.

Subsequently, another image pickup process of the image recognition application described hereinabove with reference toFIG. 8is described with reference toFIG. 9in connection with a case wherein the image recognition section102is set so as to detect a variation of the state of an image pickup object. For example, when a user wants to record image data of the upper part of the body of the user picked up when the user moves the head in front of the video camera23, the user will change the setting of the SMART CAPTURE as the image pickup application program described above so that the image recognition section102may detect a variation of the state of the image pickup object. In step S1′, the image pickup section101starts image pickup of the upper part of the body of the user and outputs resulting image data to the image recognition section102and the delaying section103. Accordingly, picked up real time image data of the upper part of the body of the user are successively inputted to the image recognition section102and the delaying section103.

In step S2′, the image recognition section102calculates a sum total of pixel values of all pixels of image data for one frame for each predetermined number of frames (for example, for each 10 frames) from among the image data inputted thereto from the image pickup section101. Then, the image recognition section102calculates a difference value of the sum total of the pixel values of all of the pixels of the image data for one frame fetched at the present point of time from the sum total of pixel values of image data for one frame fetched prior by 10 frames in time (prior by a predetermined time).

In step S3′, the image recognition section102determines whether or not the difference value calculated in step S2′ is greater than a reference value set in advance (whether or not the user has moved the head) The processes in steps S2′ and S3′ are repeated until it is determined in step S3′ that the difference value is greater than the predetermined reference value set in advance. If it is determined in step S3′ that the difference value calculated in step S2′ is greater than the predetermined reference value, then the control advances to step S4′.

In the meantime, the delaying section103delays the image data inputted thereto from the image recognition section102by a time required for the processes of the image recognition section102in steps S2′ and S3′ and outputs the delayed image data to the image fetching section104. The delaying section103adjusts its delay time in response to the information detected by the image recognition section102.

In step S4′, the image recognition section102determines, from the fact that the difference value calculated in step S2′ is greater than the predetermined reference value, that a variation of the state has occurred with the image pickup object (the user has moved the head) and generates and outputs a trigger signal to the image fetching section104. The image fetching section104fetches the image data inputted thereto from the delaying section103in synchronism with the trigger signal inputted thereto from the image recognition section102, and outputs the fetched still image data to the image storage section105. In step S5′, the image storage section105stores the still image data inputted thereto from the image fetching section104.

It is to be noted that the delaying section103adjusts its delay time also in response to the type of information detected by the image recognition section102. For example, when the image recognition section102detects an event of a variation of the state occurring with the image pickup object such as a movement of the head of the user, a trigger signal is outputted after the user has moved the head. However, it is meaningless if image data are recorded after the image of the user has disappeared from the frame. Therefore, the delay time is adjusted so that image data immediately before the user moves the head may be recorded.

In this manner, in the personal computer1of the present embodiment, recording of an image can be controlled based on arbitrary information detected from image information. Therefore, the present invention can be applied, for example, to a monitor camera for burglar prevention.

Further, while, in the personal computer1of the present embodiment, a still picture is stored in synchronism with the trigger signal, motion picture data may alternatively be recorded for a predetermined period of time in synchronism with the trigger signal.

It is to be noted that a computer program for causing a computer to execute such processes as described above can be provided to a user not only as a medium in the form of an information recording medium such as a magnetic disk or a CD-ROM but also through a network providing medium such as the Internet or a digital satellite.