MOVING OBJECT DETECTION DEVICE, CONTROL DEVICE, MOVABLE BODY, MOVING OBJECT DETECTION METHOD AND PROGRAM

A moving object detection device includes a processor and a computer-readable storage medium. The computer-readable storage medium stores a program that, when executed by the processor, causes the processor to obtain a plurality of images photographed by a camera carried by a movable body, determine movement of a photographed objects based on the plurality of images, determine movement of the movable body, and detect whether the photographed object is a moving object based on the movement of the photographed object and the movement of the movable body.

TECHNICAL FIELD

The present disclosure relates to a moving object detection device, a control device, a movable body, and a moving object detection method and program.

BACKGROUND

Japanese Patent Application Laid-Open No. 2994170 discloses a vehicle periphery monitoring device configured to detect presence of a peripheral approaching vehicle/peripheral cutting-in vehicle in an optical flow in a same direction as the moving direction of an assumed image when the peripheral approaching vehicle/peripheral cutting-in vehicle exists.

SUMMARY

Embodiments of the present disclosure provide a moving object detection device including a processor and a computer-readable storage medium. The computer-readable storage medium stores a program that, when executed by the processor, causes the processor to obtain a plurality of images photographed by a camera carried by a movable body, determine movement of a photographed objects based on the plurality of images, determine movement of the movable body, and detect whether the photographed object is a moving object based on the movement of the photographed object and the movement of the movable body.

Embodiments of the present disclosure provide a controller including a processor and a computer-readable storage medium. The computer-readable storage medium stores a program that, when executed by the processor, causes the processor to obtain a plurality of images photographed by a camera carried by a movable body, determine movement of a photographed objects based on the plurality of images, determine movement of the movable body, detect whether the photographed object is a moving object based on the movement of the photographed object and the movement of the movable body to obtain a detection result, and control a photographing condition of the camera based on the detection result.

Embodiments of the present disclosure provide a moving object detection method. The method includes obtaining a plurality of images photographed by a camera carried on a movable body, determining movement of a photographed object based on the plurality of images, determining movement of the movable body, and detecting whether the photographed object is a moving object based on the movement of the photographed object and the movement of the movable body.

REFERENCE NUMERALS

DETAILED DESCRIPTION OF THE EMBODIMENTS

The present disclosure is described through embodiments, but following embodiments do not limit the present disclosure. Not all combinations of features described in embodiments are necessary for solutions of the present disclosure. Those of ordinary skill in the art can make various modifications or improvements to following embodiments. Such modifications or improvements are within the scope of the present disclosure.

Various embodiments of the present disclosure are described with reference to flowcharts or block diagrams. In this disclosure, a block in the figures can represent (1) an execution stage of a process of operation or (2) a functional unit of a device for operation execution. The referred stage or unit can be implemented by a programmable circuit and/or a processor. A special-purpose circuit may include a digital and/or analog hardware circuit or may include an integrated circuit (IC) and/or a discrete circuit. The programmable circuit may include a reconfigurable hardware circuit. The reconfigurable hardware circuit may include logical AND, logical OR, logical XOR, logical NAND, logical NOR, other logical operation circuits, a trigger, a register, a field-programmable gate arrays (FPGA), a programmable logic array (PLA), or another storage device.

A computer-readable medium may include any tangible device that can store commands executable by an appropriate device. The commands, stored in the computer-readable medium, can be executed to perform operations consistent with the disclosure, such as those specified according to the flowchart or the block diagram described below. The computer-readable medium may include an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, etc. The computer-readable medium may include a floppy Disk®, hard drive, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray® disc, memory stick, integrated circuit card, etc.

A computer-readable command may include any one of source code or object code described by any combination of one or more programming languages. The source or object codes include traditional procedural programming languages. The traditional procedural programming languages can be assembly commands, command set architecture (ISA) commands, machine commands, machine-related commands, microcode, firmware commands, status setting data, or object-oriented programming languages and “C” programming languages or similar programming languages such as Smalltalk, JAVA (registered trademark), C++, etc. Computer-readable commands can be provided locally or via a wide area network (WAN) such as a local area network (LAN) or the Internet to a general-purpose computer, a special-purpose computer, or a processor or programmable circuit of other programmable data processing device. The processor or the programmable circuit can execute computer-readable commands to be a manner for performing the operations specified in the flowchart or block diagram. The example of the processor includes a computer processor, a processing unit, a microprocessor, a digital signal processor, a controller, a microcontroller, etc.

FIG. 1illustrates an example of an appearance of an unmanned aerial vehicle (UAV)10and a remote operation device300. The UAV10includes a UAV body20, a gimbal50, a plurality of camera devices60, and a camera device100. The gimbal50and the camera device100are an example of a camera system. The UAV10is a movable body, which includes an aerial vehicle capable of moving in the air, a vehicle capable of moving on the ground, a ship capable of moving on the water, etc. The aerial body moving in the air not only includes the UAV10but also includes other aircrafts, airships, helicopters, etc., capable of moving in the air.

The UAV body20includes a plurality of rotors. The plurality of rotors are an example of the propeller. The UAV body20controls rotations of the plurality of rotors to cause the UAV10to fly. The UAV body20uses, for example, four rotors to cause the UAV10to fly. The number of rotors is not limited to four. In some embodiments, the UAV10may also be a fixed-wing aircraft without a rotor.

The camera device100is an imaging camera that captures an object within a desired imaging range. The gimbal50can rotatably support the camera device100. The gimbal50is an example of a supporting mechanism. For example, the gimbal50uses an actuator to rotatably support the camera device100on a pitch axis. The gimbal50uses an actuator to further support the camera device100rotatably by using a roll axis and a yaw axis as rotation axes. The gimbal50can rotate the camera device100around at least one of the yaw axis, the pitch axis, or the roll axis to change an attitude of the camera device100.

The plurality of camera devices60are sensing cameras that sense surroundings to control flight of the UAV10. Two of the camera devices60may be arranged at a head, i.e., the front, of the UAV10. The other two camera devices60may be arranged at the bottom of the UAV10. The two camera devices60at the front can be used in pair, which function as a stereo camera. The two camera devices60at the bottom may also be used in pair, which function as a stereo camera. The UAV10can generate three-dimensional space data for the surrounding of the UAV10based on images captured by the plurality of camera devices60. The number of the camera devices60of the UAV10is not limited to four, and can be one. The UAV10may also include at least one camera device60at each of the head, tail, each side, bottom, and top. An angle of view that can be set in the camera device60may be larger than an angle of view that can be set in the camera device100. The camera device60may include a single focus lens or a fisheye lens.

The remote operation device300communicates with the UAV10to control the UAV10remotely. The remote operation device300may communicate with the UAV10wirelessly. The remote operation device300transmits to the UAV10instruction information indicating various commands related to the movement of the UAV10such as ascent, descent, acceleration, deceleration, forward, backward, rotation, etc. The instruction information includes, for example, instruction information to ascend the UAV10. The instruction information may indicate a desired height for the UAV10. The UAV10moves to a height indicated by the instruction information received from the remote operation device300. The instruction information may include an ascending command to ascend the UAV10. The UAV10ascends when receiving the ascending command. When the UAV10reaches an upper limit in height, even the UAV10receives the ascending command, the UAV10may be limited from further ascending.

FIG. 2illustrates an exemplary schematic diagram of a functional block of the UAV10according to some embodiments of the present disclosure. The UAV10includes a UAV controller30, a storage device37, a communication interface36, a propeller40, a global position system (GPS) receiver41, an inertia measurement unit (IMU)42, a magnetic compass43, a barometric altimeter44, a temperature sensor45, a humidity sensor46, the gimbal50, the camera device60, and the camera device100.

The communication interface36communicates with the remote operation device300and other devices. The communication interface36may receive instruction information from the remote operation device300, including various commands for the UAV controller30. The storage device37stores programs needed for the UAV controller30to control the propeller40, the GPS receiver41, the IMU42, the magnetic compass43, the barometric altimeter44, the temperature sensor45, the humidity sensor46, the gimbal50, the camera devices60, and the camera device100. The storage device32may be a computer-readable storage medium and may include at least one of SRAM, DRAM, EPROM, EEPROM, or a USB storage drive. The storage device32may be detachably arranged inside the UAV body20.

The UAV controller30controls the UAV10to fly and photograph according to the programs stored in the storage device37. The UAV controller30may include a microprocessor such as a central processing unit (CPU) or a micro processing unit (MPU), a microcontroller such as a microcontroller unit (MCU), etc. The UAV controller30controls the UAV10to fly and photograph according to the commands received from the remote operation device300through the communication interface36. The propeller40propels the UAV10. The propeller40includes a plurality of rotators and a plurality of drive motors that cause the plurality of rotors to rotate. The propeller40causes the plurality of rotors to rotate through the plurality of drive motors to cause the UAV10to fly according to the commands from the UAV controller30.

The GPS receiver41receives a plurality of signals indicating time transmitted from a plurality of GPS satellites. The GPS receiver41calculates the position (latitude and longitude) of the GPS receiver41, i.e., the position of the UAV10(latitude and longitude), based on the received plurality of signals. The IMU42detects an attitude of the UAV10. The IMU42detects accelerations of the UAV10in three axis directions of front and back, left and right, and up and down, and angular velocities in three axis directions of the pitch axis, roll axis, and yaw axis, as the attitude of the UAV10. The magnetic compass43detects an orientation of the head of the UAV10. The barometric altimeter44detects a flight altitude of the UAV10. The barometric altimeter44detects an air pressure around the UAV10, and converts the detected air pressure into an altitude to detect the altitude. The temperature sensor45detects a temperature around the UAV10. The humidity sensor46detects humidity around the UAV10.

The camera device100includes an imaging unit102and a lens unit200. The lens unit200is an example of a lens device. The imaging unit102includes an image sensor120, a camera controller110, and a storage device130. The image sensor120may be composed of a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS). The image sensor120captures an optical image imaged through a plurality of lenses210, and outputs image data of the captured optical image to the camera controller110. The camera controller110may be composed of a microprocessor such as a central processing unit (CPU), a micro processing unit (MPU), etc., or a microcontroller such as a microcontroller unit (MCU). The camera controller110can control the camera device100according to operation commands of the camera device100from the UAV controller30. The storage device130may be a computer-readable storage medium and may include at least one of SRAM, DRAM, EPROM, EEPROM, or a USB flash drive. The storage device130stores programs required for the camera controller110to control the image sensor120. The storage device130may be detachably arranged inside a housing of the camera device100.

The lens unit200includes the plurality of lenses210, a plurality of lens drivers212, and a lens controller220. The plurality of lenses210may function as a zoom lens, a varifocal lens, and a focus lens. At least some or all of the plurality of lenses210are configured to move along an optical axis. The lens unit200may be an interchangeable lens arranged to be detachable from the imaging unit102. The lens driver212causes at least some or all of the plurality of lenses210to move along the optical axis through a mechanism member such as a cam ring. The lens driver212may include an actuator. The actuator may include a step motor. The lens controller220drives the lens driver212according to lens control commands from the imaging unit102to cause one or the plurality of lenses210to move along the optical axis through the mechanism member. The lens control commands are, for example, zoom control commands and focus control commands.

The lens unit200further includes a storage device222and a position sensor214. The lens controller220controls the lens210to move in the direction of the optical axis through a lens driver212according to lens operation commands from the imaging unit102. Some or all of the lenses210move along the optical axis. The lens controller220controls at least one of the lenses210to move along the optical axis to execute at least one of a zoom operation or a focus operation. The position sensor214detects the position of the lens210. The position sensor214may detect a current zoom position or a focus position.

The lens driver212may include a vibration correction mechanism. The lens controller220can cause the lens210to move along the direction of the optical axis or perpendicular to the direction of the optical axis through the vibration correction mechanism to execute a vibration correction. The lens driver212may drive the vibration correction mechanism by a step motor to perform the vibration correction. In some embodiments, the step motor may drive the vibration correction mechanism to cause the image sensor120to move along the direction of the optical axis or the direction perpendicular to the direction of the optical axis to perform the vibration correction.

The storage device222stores control values of the plurality of lenses210moved by the lens drivers212. The storage device222may include at least one of SRAM, DRAM, EPROM, EEPROM, or a USB storage drive.

In the above-described UAV10, a moving object is detected from photographed objects of an image photographed by the camera device100. The camera device100may control exposure, focus position, and white balance based on a detection result of the moving object. The UAV10may follow the moving object based on the detection result of the moving object.

In some embodiments, the UAV controller30includes a receiver31, a setting circuit32, an acquisition circuit33, a determination circuit34, and a detection circuit35. The UAV controller30is an example of a moving object detection device for detecting the moving object.

In some embodiments, the acquisition circuit33is configured to obtain a plurality of images photographed by the camera device100carried by the UAV10. The acquisition circuit33may obtain a plurality of images continuously photographed by the camera device100. The acquisition circuit33may further obtain a plurality of images, which form a dynamic image, photographed by the camera device100.

In some embodiments, the determination circuit34is configured to determine movement of the photographed object photographed by the camera device100based on the plurality of images. The determination circuit34determines the movement of the photographed object in the images photographed by the camera device100. The determination circuit34may compare the plurality of images to determine a movement vector of the photographed object in the images as the movement of the photographed object. The determination circuit34may derive an optical flow based on the plurality of images to determine the movement of the photographed object. The determination circuit34may divide the image into a plurality of blocks to derive the movement vector according to each of the blocks to derive the optical flow. The determination circuit34may derive the movement vector according to each pixel of the image to derive the optical flow.

In some embodiments, the determination circuit34further determines movement of the UAV10. The determination circuit34may determine a speed and a moving direction of the UAV10as the movement of the UAV10. The determination circuit34may determine the movement of the UAV10based on the position of the UAV10detected by the GPS receiver41. The determination circuit34may further determine the movement of the UAV10based on information from other sensors, such as the magnetic compass43, inertia measurement unit (IMU)42, etc. The determination circuit34may further determine a distance from the camera device100to the photographed object. The determination circuit34may derive distance information according to parallax images photographed by the camera device100to determine the distance to the photographed object. The determination circuit34may further determine the movement of the camera device100relative to the gimbal50. The determination circuit34is an example of a first determination circuit, a second determination circuit, a third determination circuit, and a fourth determination circuit.

In some embodiments, the detection circuit35is configured to detect the moving object from the photographed objects in the plurality of images based on the movement of the photographed objects and the movement of the UAV10. That is, the detection circuit35can detect whether a photographed object is the moving object. The detection circuit35assumes that the photographed objects in the plurality of images are non-moving object and derive the movement of the photographed objects in the plurality of images based on the movement of the UAV10. The detection circuit35detects the moving object from the photographed objects in the plurality of images based on the derived movement of the photographed objects and the movement of the photographed objects determined by the determination circuit34. The detection circuit35may detect a photographed object having derived movement different from the movement of the photographed object determined by the determination circuit34as the moving object. When the determination circuit34determines the distance to the photographed objects, the detection circuit35may detect the moving object from the photographed objects that are within a predetermined distance range. The detection circuit35may further detect, from the photographed objects in the plurality of images, a photographed object that satisfies a predetermined size requirement of a to-be-detected moving object (also referred to as a “target moving object”) as the moving object (i.e., the photographed object is the target moving object), based on the movement of the photographed objects and the movement of the movable body. The detection circuit35may thus detect the moving object from the photographed objects in the plurality of images based on the movement of the photographed objects, the movement of the movable body, and the movement of the camera device100relative to the gimbal50.

In some embodiments, the detection circuit35may determine, from various movement vectors of the optical flow, a movement vector different from the movement vectors of the photographed objects derived based on the movement of the UAV10. The detection circuit35may detect a photographed object corresponding to the determined movement vector as the moving object.

In some embodiments, the detection circuit35may determine, from the various pixels forming the image, a pixel having at least one of a direction or an amplitude of the movement vector in the optical flow different from the movement vectors derived based on the movement of the UAV10. The detection circuit35determines a pixel group formed by adjacent pixels according to the determined pixel. When the number of pixels of the pixel group exceeds a predetermined threshold, the detection circuit35may detect the area formed by the pixel group of the image as the moving object.

In some embodiments, the detection circuit35may further determine, from blocks (e.g., 8×8 (pixels), 16×16 (pixels)) forming the image, a block having at least one of a direction or an amplitude of the movement vector in the optical flow different from the movement vectors derived based on the movement of the UAV10. The detection circuit35may determine a block group formed by adjacent blocks from determined blocks. When a number of pixels of the block group exceeds a predetermined threshold, the detection circuit35may detect the area of the image formed by the block group as the moving object.

In some embodiments, the receiver31is configured to receive the size of the to-be-detected moving object. For example, the receiver31may receive the size of the to-be-detected moving object from the user through the remote operation device300. The receiver31may receive the size of the to-be-detected moving object relative to the image photographed by the camera device100. The receiver31may further receive the image dimension (pixel quantity in the horizontal direction×pixel quantity in the vertical direction) relative to the image photographed by the camera device100as the size of the to-be-detected moving object. In this disclosure, “pixel quantity” refers to the number of pixels.

In some embodiments, when the receiver31receives the image dimension relative to the image photographed by the camera device100as the size of the to-be-detected moving object, the image dimension relative to the image may change according to the distance from the camera device100to the moving object. Therefore, when the distance from the to-be-detected moving object to the camera device100and the size of the moving object are predetermined, the receiver31may receive the size of the to-be-detected moving object through the image dimension relative to the image. In other embodiments, the actual size of the moving object may be arbitrary. When a ratio of the moving object relative to the image is predetermined, the receiver31may receive the size of the to-be-detected moving object through the image dimension relative to the image.

In some embodiments, the receiver31may further receive the actual size of the to-be-detected moving object. The receiver31may receive at least one of a width or height of the to-be-detected moving object as the actual size of the to-be-detected moving object. After the detection circuit35detects the distance to the photographed object as a candidate of the moving object, the actual size of the to-be-detected moving object may be converted to the image dimension relative to the image according to the distance.

In some embodiments, the setting circuit32sets a size condition for the to-be-detected moving object based on the size of the to-be-detected moving object received by the receiver31. The setting circuit32may set a pixel quantity of a smallest image dimension that can be used by the detection circuit35to detect a photographed object as a moving object, as the size condition of the to-be-detected moving object. The pixel quantity may be used as a threshold for the detection circuit35to detect the moving object from the photographed objects.

In some embodiments, the camera controller110may control photographing condition of the camera device based on the detection result of the moving object detected by the detection circuit35. The camera controller110may control at least one of the photographing conditions including exposure, focus position, or white balance. The camera controller110may further control at least one condition of exposure, focus position, or the white balance based on the area determined by the detection result of the moving object detected by the detection circuit35. The camera controller110may perform automatic exposure processing based on the determined area. The camera controller110may further perform automatic focus processing to focus on the determined area. The camera controller110may perform automatic white balance processing by determining a light source in the area and deriving a white balance correction value corresponding to the light source.

In some embodiments, the UAV controller30may control the flight of the UAV10to follow the moving object based on the detection result of the moving object detected by the detection circuit35.

FIG. 3andFIG. 4illustrate an example of the optical flow. The optical flow shown inFIG. 3andFIG. 4is an example of an optical flow derived from the plurality of images photographed by the camera device100facing downward during the flight of the UAV10. That is, the optical flow shown inFIG. 3andFIG. 4is an example of an optical flow derived from the plurality of images photographed by the camera device100towards a camera direction with a vertical downward component during the flight of the UAV10.

As shown inFIG. 3, when a person500as a moving object moves in a direction same as a moving direction of the UAV10at a speed different from that of the UAV10, the movement vector501of the person500has a direction and amplitude different from those of the other movement vectors502in the optical flow. The detection circuit35detects the collection of the pixels having such movement vector501as the moving object. For example, the detection circuit35detects a photographed object in a rectangular area510as the moving object.

As shown inFIG. 4, when the person500as the moving object moves in a direction opposite to the moving direction of the UAV10, the amplitude of a movement vector503of the person500is different from the amplitudes of the other movement vectors in the optical flow. The detection circuit35detects a collection of pixels having the movement vector503as the moving object. For example, the detection circuit35detects a photographed object of a rectangular area512as the moving object.

As described above, the detection circuit35is configured to take into consideration the optical flow derived from the plurality of images photographed by the camera device100and the movement of the UAV10, and determine, from the movement vectors in the optical flow, the movement vector having at least one of the amplitude or direction different from those of the movement vectors caused by the movement of the UAV10, to detect the moving object from the photographed objects in the plurality of images. When the size of the to-be-detected moving object is predetermined, the detection circuit35may effectively determine the movement vector corresponding to the to-be-detected moving object from the plurality of movement vectors in the optical flow and detect the moving object.

FIG. 5illustrates a schematic flowchart of a method of detecting a moving object according to some embodiments of the present disclosure. The UAV controller30sets the UAV10to a moving object detection mode (S100). The receiver31receives a pixel quantity threshold corresponding to the size of the to-be-detected object from the user, and the setting circuit32sets the pixel quantity threshold as a moving-object-detection threshold (S102). The UAV controller30controls the gimbal50to be fixed to fix a photographing direction of the camera device100(S104). For example, after controlling the gimbal50to cause the photographing direction of the camera device100vertically downward, the UAV controller30controls the gimbal50to be fixed to maintain the photographing direction of the camera device100. The UAV controller30controls the UAV10to start flying (S106).

In some embodiments, when the images photographed by the camera device100include a photographed object at infinity, among the movement vectors of the optical flow, there may be a movement vector that almost does not include a movement vector component following the movement of the UAV10. When such the movement vector exists, the detection circuit35may not be able to accurately detect the moving object. Therefore, the UAV controller30may control the gimbal to cause the photographing direction of the camera device100to be vertically downward and control the flight of the UAV10to cause the height of the UAV10to be maintained within a predetermined height to the ground. The UAV controller30may control the flight of the UAV10to cause the distance to the farthest photographed object (background) photographed by the camera device100to be maintained within a predetermined distance. For example, the UAV controller30may control the flight of the UAV10to cause a distance from the wall in the photographing direction of the camera device100to the UAV10to be maintained within the predetermined distance.

During the flight of the UAV10, the camera device100starts to photograph dynamic images (S108). The determination circuit34derives the optical flow based on the dynamic images (S110). The detection circuit35detects a pixel set of the movement vector of the optical flow, and the pixel set has at least one of the amplitude or direction of the movement vector different from that derived based on the moving direction of the UAV10(S112). The detection circuit35determines whether the pixel quantity of the detected pixel set is larger than or equal to the threshold (S114). When the pixel quantity of the detected pixel set is not larger than or equal to the threshold, the UAV controller30repeats the processes after process S110.

On the other hand, when the pixel quantity of the detected pixel set is larger than or equal to the threshold, the detection circuit35detects the area of the image composed of the pixel set as the moving object (S116).

In some embodiments, the detection circuit35may determine the movement vector having at least one of the amplitude or direction different from that of the movement vector of the movement of the UAV10from the movement vectors in the optical flow, such that the moving object of the desired size may be detected from the photographed objects in the plurality of images. Therefore, without predetermining the movement of the moving object, the moving object may be accurately detected from the photographed objects of the images photographed by the camera device100.

In the above, an example of fixing the photographing direction of the camera device100by fixing the gimbal50is described. In a scenario that the gimbal50is not fixed, the detection circuit35may detect the moving object by considering the photographing direction of the camera device100. In some embodiments, the detection circuit35may detect the pixel set. The pixel set includes the movement vector, among the movement vectors in the optical flow, that has at least one of the amplitude or direction different from the movement vector derived based on the moving direction of the UAV10and the moving direction of the camera device100.

FIG. 6illustrates a schematic diagram for describing hardware configuration according to some other embodiments of the present disclosure. Programs installed on the computer1200can cause the computer1200to function as an operation associated with a device or one or more units of the device according to embodiments of the present disclosure. In some embodiments, the program can cause the computer1200to implement the operation or one or more units. The program may cause the computer1200to implement a process or a stage of the process according to embodiments of the present disclosure. The program may be executed by a CPU1212to cause the computer1200to implement a specified operation associated with some or all blocks in the flowchart and block diagram described in the present specification.

In some embodiments, the computer1200includes the CPU1212and a RAM1214. The CPU1212and the RAM1214are connected to each other through a host controller1210. The computer1200further includes a communication interface1222, and an I/O unit. The communication interface1222and the I/O unit are connected to the host controller1210through an I/O controller1220. The computer1200further includes a ROM1230. The CPU1212operates according to programs stored in the ROM1230and the RAM1214to control each of the units.

The communication interface1222communicates with other electronic devices through networks. A hardware driver may store the programs and data used by the CPU1212of the computer1200. The ROM1230stores a boot program executed by the computer1200during operation, and/or the program dependent on the hardware of the computer1200. The program is provided through a computer-readable storage medium such as CR-ROM, a USB storage drive, or IC card, or networks. The program is installed in the RAM1214or the ROM1230, which can also be used as examples of the computer-readable storage medium, and is executed by the CPU1212. Information processing described in the program is read by the computer1200to cause cooperation between the program and the above-mentioned various types of hardware resources. The computer1200implements information operations or processes to constitute the device or method.

For example, when the computer1200communicates with external devices, the CPU1212can execute a communication program loaded in the RAM1214and command the communication interface1222to process the communication based on the processes described in the communication program. The CPU1212controls the communication interface1222to read transmitting data in a transmitting buffer provided by a storage medium such as the RAM1214or the USB storage drive and transmit the read transmitting data to the networks, or write data received from the networks in a receiving buffer provided by the storage medium.

The CPU1212can cause the RAM1214to read all or needed portions of files or databases stored in an external storage medium such as a USB storage drive, and perform various types of processing to the data of the RAM1214. Then, the CPU1212can write the processed data back to the external storage medium.

The CPU1212can store various types of information such as various types of programs, data, tables, and databases in the storage medium and process the information. For the data read from the RAM1214, the CPU1212can perform the various types of processes described in the present disclosure, including various types of operations, information processing, condition judgment, conditional transfer, unconditional transfer, information retrieval/replacement, etc., specified by a command sequence of the program, and write the result back to the RAM1214. In addition, the CPU1212can retrieve information in files, databases, etc., in the storage medium. For example, when the CPU1212stores a plurality of entries having attribute values of a first attribute associated with attribute values of a second attribute in the storage medium, the CPU1212can retrieve an attribute from the plurality of entries matching a condition specifying the attribute value of the first attribute, and read the attribute value of the second attribute stored in the entry. As such, the CPU1212obtains the attribute value of the second attribute associated with the first attribute that meets the predetermined condition.

The above-described programs or software modules may be stored on the computer1200or in the computer-readable storage medium near the computer1200. The storage medium such as a hard disk drive or RAM provided in a server system connected to a dedicated communication network or Internet can be used as a computer-readable storage medium. Thus, the program can be provided to the computer1200through the networks.

An execution order of various processing such as actions, sequences, processes, and stages in the devices, systems, programs, and methods shown in the claims, the specifications, and the drawings, can be any order, unless otherwise specifically indicated by “before,” “in advance,” etc., and as long as an output of previous processing is not used in subsequent processing. Operation procedures in the claims, the specifications, and the drawings are described using “first,” “next,” etc., for convenience. However, it does not mean that the operating procedures must be implemented in this order.

The present disclosure is described above with reference to embodiments, but the technical scope of the present disclosure is not limited to the scope described in the above embodiments. For those skilled in the art, various changes or improvements can be made to the above-described embodiments. It is apparent that such changes or improvements are within the technical scope of the present disclosure.