Patent Description:
Videoconference systems are now in widespread use, allowing users at remote places to hold a meeting via a communication network such as the Internet. In such videoconference systems, a communication terminal for a remote conference system is provided in a conference room where attendants of one party in a remote conference are attending. This communication terminal collects an image or video of the conference room including the attendants and sound such as speech made by the attendants, and transmits digital data converted from the collected image (video) and/or sound to the other party's terminal provided at a different conference room. Based on the transmitted digital data, the other party's terminal displays images on a display or outputs audio from a speaker in the different conference room to enable video calling. This enables to carry out a conference among remote sites, in a state close to an actual conference (see <CIT>).

In addition, for example, <CIT> (<CIT>) discloses a relay apparatus that receives image data and audio data from a first communication terminal and relays the received image and audio data to a second communication terminal. The second communication terminal requests the relay apparatus for the image data of a desired resolution depending on the resolution of a display of the own terminal. This enables to suppress unnecessary communication, thereby smoothly performing a remote conference without communication being interrupted even in a communication network having a narrow communication band.

On the other hand, for example, <CIT> discloses connecting, to a communication terminal, an image capturing device that is capable of capturing a full spherical panoramic image in real time, and transmitting the full spherical panoramic image from the image capturing device to each communication terminal of the other party. Each communication terminal sequentially converts the received full spherical panoramic image to a planar image representing a predetermined area, which is a part of the full spherical panoramic image, and displays the planar image on a display or the like.

However, there is a case that different types of image capturing device are used in plural remote sites. For example, a special image capturing device that captures a pair of hemispherical images based on which a full spherical panoramic image is generated is used in one site, while a general-purpose image capturing device that captures planar images is used in another site. In such case, the following problem will arise at the communication terminal receiving image data.

That is, when the communication terminal that has received image data obtained from the special image capturing device converts the obtained image data to a planar image and displays the planar image, the resolution of the displayed planar image decreases, because the planar image is a part of a full spherical panoramic image. This means that, when the technique as disclosed in <CIT> (<CIT>) is applied, the resolution that is lowered in order to suppress unnecessary communication is further lowered due to the conversion to the planar image. Accordingly, the resolution of the planar image displayed on a display might be considerably low. A user who views an image with such extremely low resolution on a display will have difficulty in recognizing the surroundings or attendants of the communication counterpart.

The present invention is about a communication terminal as defined in claim <NUM>, a system as defined in claim <NUM>, a communication method as defined in claim <NUM> and a carrier means as defined in claim <NUM>.

According to an aspect of the present invention, the resolution of image data requested to an image management system is controlled depending on image types. This prevents the resolution of an image displayed on a display from getting too low, and thereby preventing a user who views the image on the display from having difficulty in recognizing the surroundings or attendants of a communication counterpart.

A more complete appreciation of the embodiments and many of the attendant advantages and features thereof can be readily obtained and understood from the following detailed description with reference to the accompanying drawings, wherein:.

As used herein, the singular forms "a", "an", and "the" are intended to include the multiple forms as well, unless the context clearly indicates otherwise.

Referring to the drawings, several embodiments of the present invention are described.

With reference to <FIG>, a description is given of generating a full spherical panoramic image.

First, a description is given of an external view of an image capturing device <NUM>, with reference to <FIG>. The image capturing device <NUM> is a digital camera for capturing images from which a <NUM>-degree full spherical panoramic image is generated. <FIG> are respectively a left side view, a front view, and a plan view of the image capturing device <NUM>.

As illustrated in <FIG>, the image capturing device <NUM> has a shape such that one can hold it with one hand. Further, as illustrated in <FIG>, an imaging element 103a is provided on a front side (anterior side) of an upper section of the image capturing device <NUM>, and an imaging element 103b is provided on a back side (rear side) thereof. These imaging elements (image sensors) 103a and 103b are used in combination with optical members (e.g., fisheye lenses 102a and 102b, described later), each being capable of capturing a hemispherical image having an angle of view of <NUM> degrees or wider. Furthermore, as illustrated in <FIG>, an operation unit <NUM> such as a shutter button is provided on an opposite side of the front side of the image capturing device <NUM>.

Hereinafter, a description is given of a situation where the image capturing device <NUM> is used, with reference to <FIG> illustrates an example of how a user uses the image capturing device <NUM>. As illustrated in <FIG>, for example, the image capturing device <NUM> is used for capturing objects surrounding the user who is holding the image capturing device <NUM> in his/her hand. The imaging elements 103a and 103b illustrated in <FIG> capture the objects surrounding the user to obtain two hemispherical images.

Hereinafter, a description is given of an overview of an operation of generating the full spherical panoramic image from the image captured by the image capturing device <NUM>, with reference to <FIG> and <FIG>. <FIG> is a view illustrating a hemispherical image (front side) captured by the image capturing device <NUM>. <FIG> is a view illustrating a hemispherical image (back side) captured by the image capturing device <NUM>. <FIG> is a view illustrating an image represented by Mercator projection. The image represented by Mercator projection as illustrated in <FIG> is referred to as a "Mercator image" hereinafter. <FIG> illustrates an example of how the Mercator image covers a surface of a sphere. <FIG> is a view illustrating the full spherical panoramic image.

As illustrated in <FIG>, an image captured by the imaging element 103a is a curved hemispherical image (front side) taken through the fisheye lens 102a described later. Also, as illustrated in <FIG>, an image captured by the imaging element 103b is a curved hemispherical image (back side) taken through the fisheye lens 102b described later. The hemispherical image (front side) and the hemispherical image (back side), which is reversed by <NUM>-degree from each other, is combined by the image capturing device <NUM>. Thus, the Mercator image as illustrated in <FIG> is generated.

The Mercator image is pasted on the sphere surface using Open Graphics Library for Embedded Systems (OpenGL ES) as illustrated in <FIG>. Thus, the full spherical panoramic image as illustrated in <FIG> is generated. In other words, the full spherical panoramic image is represented as the Mercator image facing toward a center of the sphere. It should be noted that OpenGL ES is a graphic library used for visualizing two-dimensional (2D) and three-dimensional (3D) data. The full spherical panoramic image is either a still image or a movie.

One may feel strange viewing the full spherical panoramic image, because the full spherical panoramic image is an image attached to the sphere surface. To resolve this strange feeling, an image of a predetermined area, which is a part of the full spherical panoramic image, is displayed as a planar image having fewer curves. The image of the predetermined area is referred to as a "predetermined-area image" hereinafter. Hereinafter, a description is given of displaying the predetermined-area image with reference to <FIG> and <FIG>.

<FIG> is a view illustrating positions of a virtual camera IC and a predetermined area T in a case in which the full spherical panoramic image is represented as a three-dimensional solid sphere. The virtual camera IC corresponds to a position of a point of view (viewpoint) of a user who is viewing the full spherical panoramic image represented as the three-dimensional solid sphere. <FIG> is a perspective view of <FIG>. <FIG> is a view illustrating the predetermined-area image displayed on a display. In <FIG>, the full spherical panoramic image illustrated in <FIG> is represented as a three-dimensional solid sphere CS. Assuming that the generated full spherical panoramic image is the solid sphere CS, the virtual camera IC is outside of the full spherical panoramic image as illustrated in <FIG>. The predetermined area T in the full spherical panoramic image is an imaging area of the virtual camera IC. Specifically, the predetermined area T is specified by predetermined-area information indicating a position coordinate (x(rH), y(rV), angle of view α (angle)) including an angle of view of the virtual camera IC in a three-dimensional virtual space containing the full spherical panoramic image. Zooming of the predetermined area T is implemented by enlarging or reducing a range (arc) of the angle of view α. Further, zooming of the predetermined area T is implemented by moving the virtual camera IC toward or away from the full spherical panoramic image.

The predetermined-area image, which is an image of the predetermined area T illustrated in <FIG>, is displayed as an imaging area of the virtual camera IC, as illustrated in <FIG> illustrates the predetermined-area image represented by the predetermined-area information that is set by default. In another example, the predetermined-area image may be specified by an imaging area (X, Y, Z) of the virtual camera IC, i.e., the predetermined area T, rather than the predetermined-area information, i.e., the position coordinate of the virtual camera IC. A description is given hereinafter using the position coordinate (x(rH), y(rV), and an angle of view α (angle)) of the virtual camera IC.

Hereinafter, a description is given of a relation between the predetermined-area information and the predetermined area T with reference to <FIG> is a view illustrating a relation between the predetermined-area information and the predetermined area T. As illustrated in <FIG>, a center point CP of <NUM> provides the parameters (x, y) of the predetermined-area information, where <NUM> denotes a diagonal angle of view of the predetermined area T specified the angle of view α of the virtual camera IC. f is the distance from the virtual camera IC to the center point CP. L is a distance between the center point CP and a given vertex of the predetermined area T (<NUM> is a diagonal line). In <FIG>, a trigonometric function equation generally expressed by the following equation (<NUM>) is satisfied.

Hereinafter, a description is given of an overview of a configuration of an image communication system according to this embodiment with reference to <FIG> is a schematic diagram illustrating a configuration of the image communication system according to this embodiment.

As illustrated in <FIG>, the image communication system according to this embodiment includes an image capturing device 1a, an image capturing device 1b, a videoconference terminal 3a, a videoconference terminal 3d, a display 4a, a display 4d, a communication management system <NUM>, a personal computer (PC) <NUM>, an image capturing device <NUM>, and a smartphone <NUM>. The videoconference terminal 3a, the smartphone <NUM>, the PC <NUM>, and the videoconference terminal 3d communicate data with one another via a communication network <NUM> such as the Internet. The communication network <NUM> may be either a wireless network or a wired network.

The image capturing device 1a and the image capturing device 1b are each a special digital camera, which captures an image of a subject or surroundings to obtain two hemispherical images, from which a full spherical panoramic image is generated, as described above. By contrast, the image capturing device <NUM> is a general-purpose digital camera that captures an image of a subject or surroundings to obtain a general planar image.

The videoconference terminals 3a and the videoconference terminal 3d are each a terminal dedicated to videoconferencing. The videoconference terminal 3a and the videoconference terminal 3d display an image of video calling on the displays 4a and 4d, respectively, via a wired cable such as a universal serial bus (USB). The videoconference terminal 3a usually captures an image by a camera <NUM>, which is described later. However, in a case in which the videoconference terminal 3a is connected to a cradle 2a on which the image capturing device 1a is mounted, the image capturing device 1a is preferentially used. Accordingly, two hemispherical images are obtained, from which a full spherical panoramic image is generated. When a wired cable is used for connecting the videoconference terminal 3a and the cradle 2a, the cradle 2a not only enables communications between the image capturing device 1a and the videoconference terminal 3a but also supplies power with the image capturing device 1a and holds the image capturing device 1a. In this disclosure, the image capturing device 1a, the cradle 2a, the videoconference terminal 3a, and the display 4a are located at the same site A. In the site A, four users A1, A2, A3 and A4 are participating in video calling. On the other hand, the videoconference terminal 3d and the display 4d are located at the same site D. In the site D, three users D1, D2, and D3 are participating in video calling.

The communication management system <NUM> manages and controls communication of the videoconference terminal 3a, the videoconference terminal 3d, the PC <NUM> and the smartphone <NUM>. Further, the communication management system <NUM> manages types (a general image type and a special image type) of image data exchanged. Therefore, the communication management system <NUM> also operates as a communication control system. In this disclosure, the special image is a full spherical panoramic image. The communication management system <NUM> is located, for example, at a service provider that provides video communication service. In one example, the communication management system <NUM> is configured as a single computer. In another example, the communication management system <NUM> is constituted as a plurality of computers to which divided portions (functions, means, or storages) are arbitrarily allocated. In other words, the communication management system <NUM> may be implemented by a plurality of servers that operate in cooperation with one another.

The PC <NUM> performs video calling with the image capturing device <NUM> connected thereto. In this disclosure, the PC <NUM> and the image capturing device <NUM> are located at the same site C. At the site C, one user C is participating in video calling.

The smartphone <NUM> includes a display <NUM>, which is described later, and displays an image of video calling on the display <NUM>. The smartphone <NUM> includes a complementary metal oxide semiconductor (CMOS) sensor <NUM>, and usually captures an image with the CMOS sensor <NUM>. In addition, the smartphone <NUM> is also capable of obtaining data of two hemispherical images captured by the image capturing device 1b, based on which the full spherical panoramic image is generated, by wireless communication such as Wireless Fidelity (Wi-Fi) and Bluetooth (registered trademark). In a case in which wireless communication is used for obtaining data of two hemispherical images, a cradle 2b just supplies power with the image capturing device 1b and holds the image capturing device 1b. In this disclosure, the image capturing device 1b, the cradle 2b, and the smartphone <NUM> are located at the same site B. Further, in the site B, two users B1 and B2 are participating in video calling.

The videoconference terminal 3a, the videoconference terminal 3d, the PC <NUM> and the smartphone <NUM> are each an example of a communication terminal. OpenGL ES is installed in each of those communication terminals to enable each communication terminal to generate predetermined-area information that indicates a partial area of a full spherical panoramic image, or to generate a predetermined-area image from a full spherical panoramic image that is transmitted from a different communication terminal.

The arrangement of the terminals, apparatuses and users illustrated in <FIG> is just an example, and any other suitable arrangement will suffice. For example, in the site C, an image capturing device that is capable of performing image capturing for a full spherical panoramic image may be used in place of the image capturing device <NUM>. In addition, examples of the communication terminal include a digital television, a smartwatch, and a car navigation device. Hereinafter, any arbitrary one of the image capturing device 1a and the image capturing device 1b is referred to as "image capturing device <NUM>". Further, any arbitrary one of the videoconference terminal 3a and the videoconference terminal 3d is referred to as "videoconference terminal <NUM>", hereinafter. Furthermore, any arbitrary one of the display 4a and the display 4d is referred to as "display <NUM>", hereinafter.

Hereinafter, a description is given of hardware configurations of the image capturing device <NUM>, the videoconference terminal <NUM>, the communication management system <NUM>, the PC <NUM>, and the smartphone <NUM> according to this embodiment, with reference to <FIG>. Since the image capturing device <NUM> is a general-purpose camera, a detailed description thereof is omitted.

First, a description is given of a hardware configuration of the image capturing device <NUM>, with reference to <FIG> is a block diagram illustrating a hardware configuration of the image capturing device <NUM>. A description is given hereinafter of a case in which the image capturing device <NUM> is a full spherical (omnidirectional) image capturing device having two imaging elements. However, the image capturing device <NUM> may include any suitable number of imaging elements, providing that it includes at least two imaging elements. In addition, the image capturing device <NUM> is not necessarily an image capturing device dedicated to omnidirectional image capturing. Alternatively, an external omnidirectional image capturing unit may be attached to a general-purpose digital camera or a smartphone to implement an image capturing device having substantially the same function as that of the image capturing device <NUM>.

As illustrated in <FIG>, the image capturing device <NUM> includes an imaging unit <NUM>, an image processing unit <NUM>, an imaging control unit <NUM>, a microphone <NUM>, an audio processing unit <NUM>, a central processing unit (CPU) <NUM>, a read only memory (ROM) <NUM>, a static random access memory (SRAM) <NUM>, a dynamic random access memory (DRAM) <NUM>, an operation unit <NUM>, a network interface (I/F) <NUM>, a communication device <NUM>, and an antenna 117a.

The imaging unit <NUM> includes two wide-angle lenses (so-called fish-eye lenses) 102a and 102b, each having an angle of view of equal to or greater than <NUM> degrees so as to form a hemispherical image. The imaging unit <NUM> further includes the two imaging elements 103a and 103b corresponding to the wide-angle lenses 102a and 102b respectively. The imaging elements 103a and 103b each includes an image sensor such as a CMOS sensor and a charge-coupled device (CCD) sensor, a timing generation circuit, and a group of registers. The image sensor converts an optical image formed by the fisheye lenses 102a and 102b into electric signals to output image data. The timing generation circuit generates horizontal or vertical synchronization signals, pixel clocks and the like for the image sensor. Various commands, parameters and the like for operations of the imaging elements 103a and 103b are set in the group of registers.

Each of the imaging elements 103a and 103b of the imaging unit <NUM> is connected to the image processing unit <NUM> via a parallel I/F bus. In addition, each of the imaging elements 103a and 103b of the imaging unit <NUM> is connected to the imaging control unit <NUM> via a serial I/F bus such as an I2C bus. The image processing unit <NUM> and the imaging control unit <NUM> are each connected to the CPU <NUM> via a bus <NUM>. Furthermore, the ROM <NUM>, the SRAM <NUM>, the DRAM <NUM>, the operation unit <NUM>, the network I/F <NUM>, the communication device <NUM>, and the electronic compass <NUM> are also connected to the bus <NUM>.

The image processing unit <NUM> acquires image data from each of the imaging elements 103a and 103b via the parallel I/F bus and performs predetermined processing on each image data. Thereafter, the image processing unit <NUM> combines these image data to generate data of the Mercator image as illustrated in <FIG>.

The imaging control unit <NUM> usually functions as a master device while the imaging elements 103a and 103b each usually functions as a slave device. The imaging control unit <NUM> sets commands and the like in the group of registers of the imaging elements 103a and 103b via the I2C bus. The imaging control unit <NUM> receives necessary commands from the CPU <NUM>. Further, the imaging control unit <NUM> acquires status data of the group of registers of the imaging elements 103a and 103b via the I2C bus. The imaging control unit <NUM> sends the acquired status data to the CPU <NUM>.

The imaging control unit <NUM> instructs the imaging elements 103a and 103b to output the image data at a time when the shutter button of the operation unit <NUM> is pressed. The image capturing device <NUM> may display a preview image on a display (e.g., a display of the videoconference terminal 3a) or may support displaying movie. In this case, the image data are continuously output from the imaging elements 103a and 103b at a predetermined frame rate (frames per minute).

Furthermore, the imaging control unit <NUM> operates in cooperation with the CPU <NUM> to synchronize times when the imaging elements 103a and 103b output the image data. It should be noted that although in this embodiment, the image capturing device <NUM> does not include a display unit (display), the image capturing device <NUM> may include the display.

The microphone <NUM> converts sounds to audio data (signal). The audio processing unit <NUM> acquires the audio data from the microphone <NUM> via an I/F bus and performs predetermined processing on the audio data.

The CPU <NUM> controls an entire operation of the image capturing device <NUM> and performs necessary processing. The ROM <NUM> stores various programs for the CPU <NUM>. The SRAM <NUM> and the DRAM <NUM> each operates as a work memory to store programs loaded from the ROM <NUM> for execution by the CPU <NUM> or data in current processing. More specifically, the DRAM <NUM> stores image data currently processed by the image processing unit <NUM> and data of the Mercator image on which processing has been performed.

The operation unit <NUM> collectively refers to various operation keys, a power switch, the shutter button, and a touch panel having functions of both displaying information and receiving input from a user, which may be used in combination. The user operates the operation keys to input various photographing modes or photographing conditions.

The network I/F <NUM> collectively refers to an interface circuit such as a USB I/F that allows the image capturing device <NUM> to communicate data with an external media such as an SD card or an external personal computer. The network I/F <NUM> supports at least one of wired and wireless communications. The data of the Mercator image, which is stored in the DRAM <NUM>, is stored in the external media via the network I/F <NUM> or transmitted to the external device such as the videoconference terminal 3a via the network I/F <NUM>, as needed.

The communication device <NUM> communicates data with an external device such as the videoconference terminal 3a via the antenna 117a of the image capturing device <NUM> by near distance wireless communication such as Wi-Fi and Near Field Communication (NFC). The communication device <NUM> is also capable of transmitting the data of Mercator image to the external device such as the videoconference terminal 3a.

The electronic compass <NUM> calculates an orientation and a tilt (roll angle) of the image capturing device <NUM> from the Earth's magnetism to output orientation and tilt information. This orientation and tilt information is an example of related information, which is metadata described in compliance with Exif. This information is used for image processing such as image correction of captured images. Further, the related information also includes a date and time when the image is captured by the image capturing device <NUM>, and a size of the image data.

Hereinafter, a description is given of a hardware configuration of the videoconference terminal <NUM> with reference to <FIG> is a block diagram illustrating a hardware configuration of the videoconference terminal <NUM>. As illustrated in <FIG>, the videoconference terminal <NUM> includes a CPU <NUM>, a ROM <NUM>, a RAM <NUM>, a flash memory <NUM>, a solid state drive (SSD) <NUM>, a medium I/F <NUM>, an operation key <NUM>, a power switch <NUM>, a bus line <NUM>, a network I/F <NUM>, a camera <NUM>, an imaging element I/F <NUM>, a microphone <NUM>, a speaker <NUM>, an audio input/output interface <NUM>, a display I/F <NUM>, an external device connection I/F <NUM>, a near-distance communication circuit <NUM>, and an antenna 319a for the near-distance communication circuit <NUM>.

The CPU <NUM> controls an entire operation of the videoconference terminal <NUM>. The ROM <NUM> stores a control program for operating the CPU <NUM> such as an Initial Program Loader (IPL). The RAM <NUM> is used as a work area for the CPU <NUM>. The flash memory <NUM> stores various data such as a communication control program, image data, and audio data. The SSD <NUM> controls reading or writing of various data to and from the flash memory <NUM> under control of the CPU <NUM>. A hard disk drive (HDD) may be used in place of the SSD <NUM>. The medium I/F <NUM> controls reading or writing (storing) of data with respect to a recording medium <NUM> such as a flash memory. The operation key <NUM> is operated by a user to input a user instruction such as a user selection of a destination of the videoconference terminal <NUM>. The power switch <NUM> is a switch that turns on or off the power of the videoconference terminal <NUM>.

The network I/F <NUM> allows communication of data with an external device through the communication network <NUM> such as the Internet. The camera <NUM> is an example of a built-in imaging device capable of capturing a subject under control of the CPU <NUM> to obtain image data. The imaging element I/F <NUM> is a circuit that controls driving of the camera <NUM>. The microphone <NUM> is an example of a built-in audio collecting device capable of inputting audio. The audio input/output interface <NUM> is a circuit for controlling input and output of audio signals between the microphone <NUM> and the speaker <NUM> under control of the CPU <NUM>. The display I/F <NUM> is a circuit for transmitting image data to an external display <NUM> under control of the CPU <NUM>. The external device connection I/F <NUM> is an interface circuit that connects the videoconference terminal <NUM> to various external devices. The near-distance communication circuit <NUM> is a communication circuit that communicates in compliance with the NFC (registered trademark), the Bluetooth (registered trademark) and the like.

The bus line <NUM> may be an address bus or a data bus, which electrically connects various elements such as the CPU <NUM> illustrated in <FIG>.

The display <NUM> is an example of a display unit, such as a liquid crystal or organic electroluminescence (EL) display that displays an image of a subject, an operation icon, or the like. The display <NUM> is connected to the display I/F <NUM> by a cable 4c. The cable 4c may be an analog red green blue (RGB) (video graphic array (VGA)) signal cable, a component video cable, a high-definition multimedia interface (HDMI) (registered trademark) signal cable, or a digital video interactive (DVI) signal cable.

The camera <NUM> includes a lens and a solid-state imaging element that converts an image (video) of a subject to electronic data through photoelectric conversion. As the solid-state imaging element, for example, a CMOS sensor or a CCD sensor is used. The external device connection I/F <NUM> is capable of connecting an external device such as an external camera, an external microphone, or an external speaker through a USB cable or the like. In a case in which an external camera is connected, the external camera is driven in preference to the built-in camera <NUM> under control of the CPU <NUM>. Similarly, in a case in which an external microphone is connected or an external speaker is connected, the external microphone or the external speaker is driven in preference to the built-in microphone <NUM> or the built-in speaker315 under control of the CPU <NUM>.

The recording medium <NUM> is removable from the videoconference terminal <NUM>. In addition to the flash memory <NUM>, any suitable nonvolatile memory, such as an electrically erasable and programmable ROM (EEPROM), may be used, provided that it reads or writes data under control of CPU <NUM>.

Hereinafter, a description is given of hardware configurations of the communication management system <NUM> and the PC <NUM>, with reference to <FIG> is a block diagram illustrating a hardware configuration of any one of the communication management system <NUM> and the PC <NUM>. In this disclosure, both the communication management system <NUM> and the PC <NUM> are implemented by a computer. Therefore, a description is given of a configuration of the communication management system <NUM>, and the description of a configuration of the PC <NUM> is omitted, having the same or substantially the same configuration as that of the communication management system <NUM>.

The communication management system <NUM> includes a CPU <NUM>, a ROM <NUM>, a RAM <NUM>, an HD <NUM>, a hard disc drive (HDD) <NUM>, a media drive <NUM>, a display <NUM>, a network I/F <NUM>, a keyboard <NUM>, a mouse <NUM>, a compact disc rewritable (CD-RW) drive <NUM>, and a bus line <NUM>. The CPU <NUM> controls entire operation of the communication management system <NUM>. The ROM <NUM> stores programs such as an IPL to boot the CPU <NUM>. The RAM <NUM> is used as a work area for the CPU <NUM>. The HD <NUM> stores various data such as programs for the communication management system <NUM>. The HDD <NUM> controls reading and writing of data from and to the HD <NUM> under control of the CPU <NUM>. The media drive <NUM> controls reading and writing (storing) of data from and to a recording medium <NUM> such as a flash memory. The display <NUM> displays various information such as a cursor, menus, windows, characters, or images. The network I/F <NUM> enables communication of data with an external device through the communication network <NUM>. The keyboard <NUM> includes a plurality of keys to allow a user to input characters, numbers, and various instructions. The mouse <NUM> allows a user to input an instruction for selecting and executing various functions, selecting an item to be processed, or moving the cursor. The CD-RW drive <NUM> controls reading or writing of data from or to a removable recording medium <NUM> such as a CD-RW or CD-ROM. The bus line <NUM> electrically connects those parts or devices of the communication management system <NUM> to each other as illustrated in <FIG>. Examples of the bus line <NUM> include an address bus and a data bus.

Hereinafter, a description is given of hardware of the smartphone <NUM> with reference to <FIG> is a block diagram illustrating a hardware configuration of the smartphone <NUM>. As illustrated in <FIG>, the smartphone <NUM> includes a CPU <NUM>, a ROM <NUM>, a RAM <NUM>, an EEPROM <NUM>, a CMOS sensor <NUM>, an acceleration and orientation sensor <NUM>, a medium I/F <NUM>, and a global positioning system (GPS) receiver <NUM>.

The CPU <NUM> controls an entire operation of the smartphone <NUM>. The ROM <NUM> stores programs such as an IPL to boot the CPU <NUM>. The RAM <NUM> is used as a work area for the CPU <NUM>. The EEPROM <NUM> reads or writes various data such as a control program for the smartphone <NUM> under control of the CPU <NUM>. The CMOS sensor <NUM> captures an object (mainly, a user operating the smartphone <NUM>) under control of the CPU <NUM> to obtain image data. The acceleration and orientation sensor <NUM> includes various sensors such as an electromagnetic compass for detecting geomagnetism, a gyrocompass, or an acceleration sensor. The medium I/F <NUM> controls reading or writing of data with respect to a recording medium <NUM> such as a flash memory. The GPS receiver <NUM> receives a GPS signal from a GPS satellite.

The smartphone <NUM> further includes a far-distance communication circuit <NUM>, a camera <NUM>, an imaging element I/F <NUM>, a microphone <NUM>, a speaker <NUM>, an audio input/output interface <NUM>, a display <NUM>, an external device connection I/F <NUM>, a near-distance communication circuit <NUM>, an antenna 919a for the near-distance communication circuit <NUM>, and a touch panel <NUM>.

The far-distance communication circuit <NUM> is a circuit that communicates with other device through the communication network <NUM>. The camera <NUM> is an example of a built-in imaging device capable of capturing a subject under control of the CPU <NUM> to obtain image data. The imaging element I/F <NUM> is a circuit that controls driving of the camera <NUM>. The microphone <NUM> is an example of a built-in audio collecting device capable of inputting audio. The audio input/output interface <NUM> is a circuit for controlling input and output of audio signals between the microphone <NUM> and the speaker <NUM> under control of the CPU <NUM>. The display <NUM> is an example of a display unit, such as a liquid crystal or organic electroluminescence (EL) display that displays an image of a subject, an operation icon, or the like. The external device connection I/F <NUM> is an interface circuit that connects the smartphone <NUM> to various external devices. The near-distance communication circuit <NUM> is a communication circuit that communicates in compliance with the NFC, the Bluetooth and the like. The touch panel <NUM> is an example of an input device to operate a smartphone <NUM> by touching a screen of the display <NUM>.

The smartphone <NUM> further includes a bus line <NUM>. Examples of the bus line <NUM> include an address bus and a data bus, which electrically connects the elements such as the CPU <NUM>.

It should be noted that a recording medium such as a CD-ROM or a hard disk storing any one of the above-described programs may be distributed domestically or overseas as a program product.

Hereinafter, a description is given of a functional configuration of the image communication system according to this embodiment, with reference to <FIG>. <FIG> and <FIG> are block diagrams, each illustrating a part of a functional configuration of the image communication system.

As illustrated in <FIG>, the image capturing device 1a includes an acceptance unit 12a, an image capturing unit 13a, an audio collecting unit 14a, a communication unit 18a, and a data storage/read unit 19a. These units are functions or means that are implemented by or that are caused to function by operating any of the elements illustrated in <FIG> in cooperation with the instructions of the CPU <NUM> according to the image capturing device control program expanded from the SRAM <NUM> to the DRAM <NUM>.

The image capturing device 1a further includes a memory 1000a, which is implemented by the ROM <NUM>, the SRAM <NUM>, and the DRAM <NUM> illustrated in <FIG>. The memory 1000a stores therein a globally unique identifier (GUID) identifying the own device (i.e., the image capturing device 1a).

The image capturing device 1b includes an acceptance unit 12b, an image capturing unit 13b, an audio collecting unit 14b, a communication unit 18b, a data storage/read unit 19b, and a memory 1000b. These functional units of the image capturing device 1b implement the similar or substantially the similar functions as those of the acceptance unit 12a, the image capturing unit 13a, the audio collecting unit 14a, the communication unit 18a, the data storage/read unit 19a, and the memory <NUM> of the image capturing device 1a, respectively. Therefore, redundant descriptions thereof are omitted below.

Hereinafter, referring to <FIG> and <FIG>, a further detailed description is given of each functional unit of the image capturing device 1a according to the embodiment.

The acceptance unit 12a of the image capturing device 1a is implemented by the operation unit <NUM> illustrated in <FIG>, when operating under control of the CPU <NUM>. The acceptance unit 12a receives an instruction input from the operation unit <NUM> according to a user operation.

The image capturing unit 13a is implemented by the imaging unit <NUM>, the image processing unit <NUM>, and the imaging control unit <NUM>, illustrated in <FIG>, when operating under control of the CPU <NUM>. The image capturing unit 13a captures an image of a subject or surroundings to obtain captured-image data.

The audio collecting unit 14a is implemented by the microphone <NUM> and the audio processing unit <NUM> illustrated in <FIG>, when operating under control of the CPU <NUM>. The audio collecting unit 14a collects sounds around the image capturing device 1a.

The communication unit 18a, which is implemented by instructions of the CPU <NUM>, communicates data with a communication unit <NUM> of the videoconference terminal 3a using the near-distance wireless communication technology in compliance with such as NFC, Bluetooth, or Wi-Fi.

The data storage/read unit 19a, which is implemented by instructions of the CPU <NUM> illustrated in <FIG>, stores data or information in the memory 1000a or reads out data or information from the memory 1000a.

As illustrated in <FIG>, the videoconference terminal 3a includes a data exchange unit 31a, an acceptance unit 32a, an image and audio processor 33a, a display control 34a, a determination unit 35a, a generator 36a, a change unit 37a, a communication unit 38a, and a data storage/read unit 39a. These units are functions that are implemented by or that are caused to function by operating any of the elements illustrated in <FIG> in cooperation with the instructions of the CPU <NUM> according to the control program for the videoconference terminal 3a, expanded from the flash memory <NUM> to the RAM <NUM>.

The videoconference terminal 3a further includes a memory 3000a, which is implemented by the ROM <NUM>, the RAM <NUM>, and the flash memory <NUM> illustrated in <FIG>. The memory 3000a stores an image type management DB 3001a, an image capturing device management DB 3002a, a display layout management DB 3003a, and a required resolution management DB 3004a. Among these DBs, the image type management DB 3001a is implemented by an image type management table illustrated in <FIG>. The image capturing device management DB 3002a is implemented by an image capturing device management table illustrated in <FIG>. The display layout management DB 3003a is implemented by a display layout management table illustrated in <FIG>. The required resolution management DB 3004a is implemented by required resolution management tables illustrated in <FIG>.

The videoconference terminal 3d includes a data exchange unit 31d, an acceptance unit 32d, an image and audio processor 33d, a display control 34d, a determination unit 35d, a generator 36d, a change unit 37d, a communication unit 38d, and a data storage/read unit 39d, and a memory 3000d. These functional units of the videoconference terminal 3d implement the similar or substantially the similar functions as those of the data exchange unit 31a, the acceptance unit 32a, the image and audio processor 33a, the display control 34a, the determination unit 35a, the generator 36a, the change unit 37a, the communication unit 38a, the data storage/read unit 39a, and the memory 3000a of the videoconference terminal 3a, respectively. Therefore, redundant descriptions thereof are omitted below. In addition, the memory 3000d of the videoconference terminal 3d stores an image type management DB 3001d, and an image capturing device management DB 3002d, a display layout management DB 3003d, and a required resolution management DB 3004d. These DBs 3001d, 3002d, 3003d and 3004d have the same or the substantially the same data structure as the image type management DB 3001a, the image capturing device management DB 3002a, the display layout management DB 3003a, and the required resolution management DB 3004a of the videoconference terminal 3a, respectively. Therefore, redundant descriptions thereof are omitted below.

<FIG> is a conceptual diagram illustrating the image type management table according to an embodiment of this disclosure. The image type management table stores an image data identifier (ID), an IP address, which is an example of an address of a sender terminal, and a source name, in association with one another. The image data ID is one example of image data identification information for identifying image data to be used in video communication. An identical image data ID is assigned to image data transmitted from the same sender terminal. Accordingly, a destination terminal (that is, a communication terminal that receives image data) can identify a sender terminal from which the image data is received. The IP address of the sender terminal, which is associated with specific image data ID, is an IP address of a communication terminal that transmits image data identified by that image data ID associated with the IP address. The source name, which is associated with specific image data ID, is a name for identifying an image capturing device that outputs the image data identified by that image data ID associated with the source name. The source name is one example of image type information. This source name is generated by a communication terminal such as the videoconference terminal 3a according to a predetermined naming rule.

The example of the image type management table illustrated in <FIG> indicates that four communication terminals, whose IP addresses are respectively "<NUM>. <NUM>", "<NUM>. <NUM>", "<NUM>. <NUM>", and "<NUM>. <NUM>" transmit image data identified by the image data ID "RS001", "RS002", "RS003", and "RS004", respectively. Further, according to the image type management table illustrated in <FIG>, the image types represented by the source names of those four communication terminals are "Video_Theta", "Video_Theta", "Video", and "Video", which indicate that the image types are the special image, special image, general image, and general image, respectively. In this disclosure, the special image is a full spherical panoramic image, and the general image is a planar image.

In another example, information regarding data other than the image data may be stored in the image type management table in association with the image data ID. Examples of the data other than the image data include audio data and presentation material data shared on a screen, for example, in video communication in which the image is shared. Further, information regarding data other than the image data may be stored in the image type management table in association with the image data ID. Examples of the data other than the image data include audio data and presentation material data shared on a screen, for example, in video communication in which the image is shared.

<FIG> is a conceptual diagram illustrating the image capturing device management table according to an embodiment of this disclosure. The image capturing device management table stores a vendor ID and a product ID from among the GUIDs of an image capturing device that is capable of obtaining two hemispherical images, from which a full spherical panoramic image is generated. As the GUID, a combination of a vendor ID (VID) and a product ID (PID) used in a USB device is used, for example. Those vendor ID and product ID are stored in a communication terminal such as a videoconference terminal before shipment. In another example, those IDs are added and stored in the videoconference terminal after shipment.

<FIG> are conceptual diagrams, each illustrating the display layout management table. <FIG> illustrates a data structure of the display layout management DB 3003a stored in the memory 3000a in the videoconference terminal 3a.

<FIG> illustrates a data structure of a display layout management DB <NUM> stored in the memory <NUM> of the smartphone <NUM>. <FIG> illustrates a data structure of a display layout management DB <NUM> stored in the memory <NUM> of the PC <NUM>. <FIG> illustrates a data structure of the display layout management DB 3003d stored in the memory 3000d of the videoconference terminal 3d.

The display layout management table illustrated in <FIG> stores, for each layout number indicating a layout of the plurality of display areas on the display 4a, an image data ID for identifying image data of an image to be displayed in the display area according to the layout, and a display size (number of pixels in horizontal and vertical directions) of the display area. The layout number in this table specifies one of a plurality of display areas in which the image data is to be displayed. More specifically, the layout number in this example corresponds to a number indicated at the lower right corner of each display area as illustrated in <FIG>. The display size is information acquired from the display 4a by the videoconference terminal 3a. Accordingly, when the display 4a to be used is changed, the display size in the display layout management table is also changed. In addition, when images to be displayed in the display areas are changed (switched), the image data IDs in the display layout management table are also changed (switched). For example, in <FIG>, when the user B1 starts to speak, an image of the site B that was displayed in a display area "<NUM>" is switched to be displayed in a display area "<NUM>", and an image of the site A that was displayed in the display area "<NUM>" is switched to be displayed in the display area "<NUM>". This change is performed by the change unit 37a based on audio data and the image data ID.

<FIG> are conceptual diagrams illustrating the required resolution management tables for the different image types. The required resolution management tables of <FIG> store, for respective ones of image type information indicating the different image types, the display size (number of pixels in horizontal and vertical directions) of each of the plurality of display areas of the display, in association with a required resolution (number of pixels in the horizontal and vertical directions) of image data to be requested for the communication management system <NUM>. <FIG> is an example of the required resolution management table for the special image as the image type information. <FIG> illustrates an example of the required resolution management table for the general image as the image type information. As seen from <FIG>, for the same display size (for example, <NUM>×<NUM> pixels), the required resolution (<NUM>×<NUM> pixels) for the special image is higher than the required resolution (<NUM>×<NUM> pixels) for the general image. This prevents the resolution of an image representing one site displayed on a display of another site from getting extremely low, even when the image displayed on the other site is a predetermined-area image as illustrated in <FIG>, which is an image of a predetermined area T as a partial area of the full spherical panoramic image, as illustrated in <FIG>. Accordingly, the predetermined-area image is made suitable for viewing.

Hereinafter, referring to <FIG> and <FIG>, a further detailed description is given of each functional unit of the videoconference terminal 3a.

The data exchange unit 31a of the videoconference terminal 3a is implemented by the network I/F <NUM> illustrated in <FIG>, when operating under control of the CPU <NUM>. The data exchange unit 31a exchanges data or information with communication management system <NUM> via the communication network <NUM>.

The acceptance unit 32a is implemented by the operation key <NUM>, when operating under control of the CPU <NUM>. The acceptance unit 32a receives selections or inputs from a user. An input device such as a touch panel may be used as an alternative to or in place of the operation key <NUM>.

The image and audio processor 33a, which is implemented by instructions of the CPU <NUM> illustrated in <FIG>, processes image data that is obtained by capturing a subject by the camera <NUM>. In addition, after the audio of the user is converted to an audio signal by the microphone <NUM>, the image and audio processor 33a processes audio data according to this audio signal.

Further, the image and audio processor 33a processes image data received from other communication terminal based on the image type information such as the source name, to enable the display control <NUM> to control the display <NUM> to display an image based on the processed image data. More specifically, when the image type information indicates a special image, the image and audio processor 33a converts the image data such as hemispherical image data as illustrated in <FIG> into full spherical image data to generate full spherical panoramic image data as illustrated in <FIG>, and further generates a predetermined-area image as illustrated in <FIG>. Furthermore, the image and audio processor 33a outputs, to the speaker <NUM>, an audio signal according to audio data that is received from the other communication terminal via the communication management system <NUM>. The speaker <NUM> outputs audio based on the audio signal.

The display control 34a is implemented by the display I/F <NUM>, when operating under control of the CPU <NUM>. The display control 34a controls the display <NUM> to display images or characters.

The determination unit 35a, which is implemented by instructions of the CPU <NUM>, determines an image type according to the image data received from such as the image capturing device 1a.

The generator 36a is implemented by instructions of the CPU <NUM>. The generator 36a generates a source name, which is one example of the image type information, according to the above-described naming rule, based on a determination result generated by the determination unit 35a indicating a general image or a special image (that is, full spherical panoramic image in this disclosure). For example, when the determination unit 35a determines the image type as a general image, the generator 36a generates the source name "Video" indicating a general image. By contrast, when the determination unit 35a determines the image type as a special image, the generator 36a generates the source name "Video_Theta" indicating a special image.

The change unit 37a, which is implemented by instruction of the CPU <NUM>, changes (switches) the image data IDs associated with the layout numbers in the display layout management table. More specifically, the change unit 37a changes association of a layout number with an image data ID, such that the image data ID transmitted from the site where a user who spoke immediately before is present is associated with the layout number "<NUM>". At the same time, the change unit 37a associates the image data ID that was associated with the layout number "<NUM>" before this change with a layout number that was associated with the image data ID transmitted from the site where the user who spoke immediately before is present. With this process by the change unit 37a, an image according to image data from a site where a user who spoke immediately before is present is constantly displayed the largest, as illustrated in <FIG>.

The communication unit 38a is implemented by the near-distance communication circuit <NUM> and the antenna 319a, when operating under control of the CPU <NUM>. The communication unit 38a communicates with the communication unit 18a of the image capturing device 1a using the near-distance communication technology in compliance with such as NFC, Bluetooth, or Wi-Fi. Although in the above description the communication unit 38a and the data exchange unit 31a have separate communication units, alternatively a shared communication unit may be used.

The data storage/read unit 39a, which is implemented by instructions of the CPU <NUM> illustrated in <FIG>, stores data or information in the memory <NUM> or reads out data or information from the memory <NUM>.

Hereinafter, referring to <FIG> and <FIG>, a detailed description is given of each functional unit of the communication management system <NUM>. The communication management system <NUM> includes a data exchange unit <NUM>, a resolution controller <NUM>, a determination unit <NUM>, a generator <NUM>, and a data storage/read unit <NUM>. These units are functions that are implemented by or that are caused to function by operating any of the elements illustrated in <FIG> in cooperation with the instructions of the CPU <NUM> according to the control program for the communication management system <NUM>, expanded from the HD <NUM> to the RAM <NUM>.

The communication management system <NUM> further includes a memory <NUM>, which is implemented by the RAM <NUM> and the HD <NUM> illustrated in <FIG>. The memory <NUM> stores a session management DB <NUM>, an image type management DB <NUM>, and a transmission resolution management DB <NUM>. The session management DB <NUM> is implemented by a session management table illustrated in <FIG>. The image type management DB <NUM> is implemented by an image type management table illustrated in <FIG>. The transmission resolution management DB <NUM> is implemented by a transmission resolution management table illustrated in <FIG>.

<FIG> is a conceptual diagram illustrating the session management table according to an embodiment of this disclosure. The session management table stores a session ID and an IP address of participating communication terminal, in association with each other. The session ID is one example of session identification information for identifying a session that implements video calling. The session ID is generated for each virtual conference room. The session ID is also stored in each communication terminal such as the videoconference terminal 3a. Each communication terminal selects a desired session ID from the session ID or IDs stored therein. The IP address of participating communication terminal indicates an IP address of the communication terminal participating in a virtual conference room identified by the associated session ID.

<FIG> is a conceptual diagram illustrating the image type management table according to an embodiment of this disclosure. The image type management table illustrated in <FIG> stores, in addition to the information items stored in the image type management table illustrated in <FIG>, the same session ID as the session ID stored in the session management table, in association with one another. The example of the image type management table illustrated in <FIG> indicates that three communication terminals whose IP addresses are respectively "<NUM>. <NUM>", "<NUM>. <NUM>", and "<NUM>. <NUM>" are participating in the virtual conference room identified by the session ID "se101". The communication management system <NUM> stores the same image data ID, IP address of the sender terminal, and image type information as those stored in a communication terminal such as the videoconference terminal 3a in order to transmit such information as the image type information to both a communication terminal that is already in video calling and a newly participating communication terminal that enters the virtual conference room after the video calling has started. Accordingly, the communication terminal that is already in the video calling and the newly participating communication terminal do not have to exchange such information as the image type information with each other.

<FIG> is a conceptual diagram illustrating the transmission resolution management table according to an embodiment of this disclosure. The transmission resolution management table stores an IP address of a destination (a required resolution sender terminal) to which image data is to be transmitted in a column direction in association with an IP address of an image data sender terminal from which image data is received in a row direction. Based on this table, when relaying image data received from the site A (videoconference terminal 3a) to the site D (videoconference terminal 3d) for example, the communication management system <NUM> transmits the image data having the resolution of <NUM>×<NUM> pixels to the site D.

Hereinafter, referring to <FIG> and <FIG>, a detailed description is given of each functional unit of the communication management system <NUM>.

The data exchange unit <NUM> of the communication management system <NUM> is implemented by the network I/F <NUM> illustrated in <FIG>, when operating under control of the CPU <NUM>. The data exchange unit <NUM> exchanges data or information with the videoconference terminal 3a, the videoconference terminal 3d, or the PC <NUM> via the communication network <NUM>.

The resolution controller <NUM>, which is implemented by instructions of the CPU <NUM>, controls (for example, changes) the resolution of image data transmitted from one communication terminal, based on the transmission resolution management DB <NUM>.

The determination unit <NUM>, which is implemented by instructions of the CPU <NUM>, performs various determinations.

The generator <NUM>, which is implemented by instructions of the CPU <NUM>, generates the image data ID.

The data storage/read unit <NUM> is implemented by the HDD <NUM> illustrated in <FIG>, when operating under control of the CPU <NUM>. The data storage/read unit <NUM> stores data or information in the memory <NUM> or reads out data or information from the memory <NUM>.

Hereinafter, referring to <FIG> and <FIG>, a detailed description is given of a functional configuration of the PC <NUM>. The PC <NUM> has the same or substantially the same functions as those of the videoconference terminal 3a. In other words, as illustrated in <FIG>, the PC <NUM> includes a data exchange unit <NUM>, an acceptance unit <NUM>, an image and audio processor <NUM>, a display control <NUM>, a determination unit <NUM>, a generator <NUM>, a change unit <NUM>, a communication unit <NUM>, and a data storage/read unit <NUM>. These units are functions that are implemented by or that are caused to function by operating any of the hardware elements illustrated in <FIG> in cooperation with the instructions of the CPU <NUM> according to the control program for the PC <NUM>, expanded from the HD <NUM> to the RAM <NUM>.

The PC <NUM> further includes a memory <NUM>, which is implemented by the ROM <NUM>, the RAM <NUM> and the HD <NUM> illustrated in <FIG>. The memory <NUM> stores an image type management DB <NUM>, an image capturing device management DB <NUM>, a display layout management DB <NUM>, and a required resolution management DB <NUM>. These DBs <NUM>, <NUM>, <NUM> and <NUM> have the same or the substantially the same data structure as the image type management DB 3001a, the image capturing device management DB 3002a, the display layout management DB 3003a, and the required resolution management DB 3004a of the videoconference terminal 3a, respectively. Therefore, redundant descriptions thereof are omitted below.

The data exchange unit <NUM> of the PC <NUM>, which is implemented by the network I/F <NUM>, when operating under control of the CPU <NUM> illustrated in <FIG>, implements the similar or substantially the similar function to that of the data exchange unit 31a.

The acceptance unit <NUM>, which is implemented by the keyboard <NUM> and the mouse <NUM>, when operating under control of the CPU <NUM>, implements the similar or substantially the similar function to that of the acceptance unit 32a. The image and audio processor <NUM>, which is implemented by instructions of the CPU <NUM>, implements the similar or substantially the similar function to that of the image and audio processor 33a. The display control <NUM>, which is implemented by instructions of the CPU <NUM>, implements the similar or substantially the similar function to that of the display control 34a. The determination unit <NUM>, which is implemented by instructions of the CPU <NUM>, implements the similar or substantially the similar function to that of the determination unit 35a. The generator <NUM>, which is implemented by instructions of the CPU <NUM>, implements the similar or substantially the similar function to that of the generator 36a. The change unit <NUM>, which is implemented by instructions of the CPU <NUM>, implements the similar or substantially the similar function to that of the change unit 37a. The communication unit <NUM>, which is implemented by instructions of the CPU <NUM>, implements the similar or substantially the similar function to that of the communication unit 38a. The data storage/read unit <NUM>, which is implemented by instructions of the CPU <NUM>, stores data or information in the memory <NUM> or reads out data or information from the memory <NUM>.

Hereinafter, referring to <FIG> and <FIG>, a detailed description is given of a functional configuration of the smartphone <NUM>. The smartphone <NUM> has the same or substantially the same functions as those of the videoconference terminal 3a. In other words, as illustrated in <FIG>, the smartphone <NUM> includes a data exchange unit <NUM>, an acceptance unit <NUM>, an image and audio processor <NUM>, a display control <NUM>, a determination unit <NUM>, a generator <NUM>, a change unit <NUM>, a communication unit <NUM>, and a data storage/read unit <NUM>. These units are functions that are implemented by or that are caused to function by operating any of the hardware elements illustrated in <FIG> in cooperation with the instructions of the CPU <NUM> according to the control program for the smartphone <NUM> expanded from the EEPROM <NUM> to the RAM <NUM>.

The smartphone <NUM> further includes a memory <NUM>, which is implemented by the ROM <NUM>, the RAM <NUM>, and the EEPROM <NUM> illustrated in <FIG>. The memory <NUM> stores an image type management DB <NUM>, an image capturing device management DB <NUM>, a display layout management DB <NUM>, and a required resolution management DB <NUM>. These DBs <NUM>, <NUM>, <NUM> and <NUM> have the same or the substantially the same data structure as the image type management DB 3001a, the image capturing device management DB 3002a, the display layout management DB 3003a, and the required resolution management DB 3004a of the videoconference terminal 3a, respectively. Therefore, redundant descriptions thereof are omitted below.

The data exchange unit <NUM> of the smartphone <NUM>, which is implemented by the far-distance communication circuit <NUM> illustrated in the <FIG>, when operating under control of the CPU <NUM>, implements the similar or substantially the similar function to that of the data exchange unit 31a.

The acceptance unit <NUM>, which is implemented by the touch panel <NUM>, when operating under control of the CPU <NUM>, implements the similar or substantially the similar function to that of the acceptance unit 32a.

The image and audio processor <NUM>, which is implemented by instructions of the CPU <NUM>, implements the similar or substantially the similar function to that of the image and audio processor 33a.

The display control <NUM>, which is implemented by instructions of the CPU <NUM>, implements the similar or substantially the similar function to that of the display control 34a.

The determination unit <NUM>, which is implemented by instructions of the CPU <NUM>, implements the similar or substantially the similar function to that of the determination unit 35a.

The generator <NUM>, which is implemented by instructions of the CPU <NUM>, implements the similar or substantially the similar function to that of the generator 36a.

The change unit <NUM>, which is implemented by instructions of the CPU <NUM>, implements the similar or substantially the similar function to that of the change unit 37a.

The communication unit <NUM>, which is implemented by instructions of the CPU <NUM>, implements the similar or substantially the similar function to that of the communication unit 38a.

The data storage/read unit <NUM>, which is implemented by instructions of the CPU <NUM>, stores data or information in the memory <NUM> or reads out data or information from the memory <NUM>.

Referring to <FIG>, a description is given hereinafter of operation according to the present embodiment. Firstly, a process of participating in a specific communication session is described with reference to <FIG> is a sequence diagram illustrating an operation of participating in the specific communication session. <FIG> is a view illustrating a selection screen for accepting selection of a desired communication session (virtual conference).

First, the acceptance unit 32a of the videoconference terminal 3a accepts an instruction to display the selection screen for the communication session (virtual conference room), which is input by a user (e.g., the user A1) at the site A. Then, the display control 34a controls the display 4a to display the selection screen as illustrated in <FIG> (S21). The selection screen displays selection buttons b1, b2, and b3, which respectively represent virtual conference rooms R1, R2, R3, each being a selection target. Each of the selection buttons b1, b2, and b3 is associated with the session ID.

When the user A1 selects a desired selection button (in this example, the selection button b1) on the selection screen, the acceptance unit 32a accepts selection of a communication session (S22). Then, the data exchange unit 31a transmits a request for participating in a virtual conference room to the communication management system <NUM> (S23). This participation request includes the session ID identifying the communication session for which the selection is accepted at S22, and the IP address of the videoconference terminal 3a as a request sender terminal. The communication management system <NUM> receives the participation request at the data exchange unit <NUM>.

Next, the data storage/read unit <NUM> performs a process for enabling the videoconference terminal 3a to participate in the communication session (S24). More specifically, in the session management DB <NUM> (<FIG>), the data storage/read unit <NUM> adds the IP address that is received at S23 to a field of the participating terminal IP address in a record of the same session ID as the session ID that is received at S23. The data exchange unit <NUM> transmits a response to the participation request to the videoconference terminal 3a (S25). This response to the participation request includes the session ID that is received at S23, and a result of the participation process. The videoconference terminal 3a receives the response to the participation request at the data exchange unit 31a. The following describes a case in which the process for enabling the videoconference terminal 3a to participate in the communication session is successfully completed.

Hereinafter, referring to <FIG>, a description is given of a management process of the image type information. <FIG> is a sequence diagram illustrating an operation of managing the image type information.

First, when a user (e.g., the user A1) at the site A connects the cradle 2a, on which the image capturing device 1a is mounted, to the videoconference terminal 3a, using the wired cable such as a USB cable, the data storage/read unit 19a of the image capturing device 1a reads out the GUID of the own device (e.g., the image capturing device 1a) from the memory 1000a. Then, the communication unit 18a transmits the own device's GUID to the communication unit 38a of the videoconference terminal <NUM> (S51). The videoconference terminal 3a receives the GUID of the image capturing device 1a at the communication unit 38a.

Next, the determination unit 35a of the videoconference terminal 3a determines whether the same vendor ID and product ID as those of the GUID received at S51 are stored in the image capturing device management DB 3002a (<FIG>) to determine the image type (S52). More specifically, the determination unit 35a determines that the image capturing device 1a is an image capturing device that captures a special image (a full spherical panoramic image, in this disclosure), based on determination that the same vender ID and product ID are stored in the image capturing device management DB 3002a. By contrast, the determination unit 35a determines that the image capturing device 1a is an image capturing device that captures a general image, based on determination that the same vender ID and product ID are not stored in the image capturing device management DB 3002a.

Next, the data storage/read unit 39a stores, in the image type management DB 3001a (<FIG>), the IP address of the own terminal (i.e., videoconference terminal 3a) as the sender terminal in association with the image type information, which is a determination result determined at S52 (S53). In this state, the image data ID is not yet associated. Examples of the image type information include the source name that is determined according to a predetermined naming rule, and the image type (general image or special image type).

Then, the data exchange unit 31a transmits a request for adding the image type information to the communication management system <NUM> (S54). This request for adding image type information includes the IP address of the own terminal as a sender terminal, and the image type information, both being stored at S53 in association with each other. The communication management system <NUM> receives the request for adding the image type information at the data exchange unit <NUM>.

Next, the data storage/read unit <NUM> of the communication management system <NUM> searches the session management DB <NUM> (<FIG>) using the IP address of the sender terminal received at S54 as a search key, to read out the session ID associated with the IP address (S55).

Next, the generator <NUM> generates a unique image data ID (S56). Then, the data storage/read unit <NUM> stores, in the image type management DB <NUM> (<FIG>), a new record associating the session ID that is read out at S55, the image data ID generated at S56, the IP address of the sender terminal and the image type information that are received at S54, with one another (S57). The data exchange unit <NUM> transmits the image data ID generated at S56 to the videoconference terminal 3a (S58). The videoconference terminal 3a receives the image data ID at the data exchange unit 31a.

Next, the data storage/read unit 39a of the videoconference terminal 3a stores, in the image type management DB 3001a (<FIG>), the image data ID received at S58, in association with the IP address of the own terminal (i.e., videoconference terminal 3a) as the sender terminal and the image type information that are stored at S53 (S59).

Further, the data exchange unit <NUM> of the communication management system <NUM> transmits a notification of addition of the image type information to other communication terminal (the videoconference terminal 3d, in this example) (S60). This notification of addition of the image type information includes the image data ID generated at S56, and the IP address of the own terminal (i.e., videoconference terminal 3a) as the sender terminal and the image type information that are stored at S53. The videoconference terminal 3d receives the notification of addition of the image type information at the data exchange unit 31a. The destination to which the data exchange unit <NUM> transmits the notification is other IP address that is associated with the same session ID as that associated with the IP address of the videoconference terminal 3a in the session management DB <NUM> (<FIG>). In other words, the destination is other communication terminal that is in the same virtual conference room as the videoconference terminal 3a.

Next, the data storage/read unit 39d of the videoconference terminal 3d stores, in the image type management DB 3001d (<FIG>), a new record associating the image data ID, the IP address of the sender terminal, and the image type information, which are received at S60 (S61). In substantially the same manner, the notification of addition of the image type information is transmitted to the PC <NUM> and the smartphone <NUM>, each being other communication terminal. The PC <NUM> and the smartphone <NUM> each stores the image data ID, the IP address of the sender terminal, and the image type information, in corresponding one of the image type management DBs <NUM> and <NUM>. Through the operation as described heretofore, the same information is shared among the communication terminals in the image type management DBs 3001a, 3001d, <NUM> and <NUM>, respectively.

Hereinafter, referring to <FIG>, a description is given of a resolution request process. <FIG> is a sequence diagram illustrating an operation of requesting the resolution.

As illustrated in <FIG>, the data storage/read unit 39a of the videoconference terminal 3d reads out each information item of the image data ID and the display size from the display layout management DB 3003d (<FIG>) (S71). Then, the data storage/read unit 39a searches the image type management DB 3001d (<FIG>) using the image data ID that is read out at S71 as a search key, to read out the image type information associated with the image data ID (S72). Next, the data storage/read unit 39a selects the table of <FIG> or the table of <FIG> in the required resolution management DB 3004d based on the image type information that is read out at S72. Further, the data storage/read unit 39a searches the selected table using the display size that is read out at S71 as a search key to read out information indicating the required resolution associated with the display size (S73).

Next, the data exchange unit 31d of the videoconference terminal 3d transmits resolution request information indicating a request of the resolution to the communication management system <NUM> (S74). This resolution request information contains the image data ID that is read out at S71 and the required resolution that is read out at S73. The communication management system <NUM> receives the resolution request information at the data exchange unit <NUM>.

Next, the data storage/read unit <NUM> of the communication management system <NUM> searches the image type management DB <NUM> (<FIG>) using the image data ID received at S74 as a search key, to read out the IP address of the sender terminal associated with the image data ID (S75). Then, the data storage/read unit <NUM> stores the required resolution received at S75 in accordance with the IP address that is read out at S75 in the transmission resolution management DB <NUM> (S76).

A description is given heretofore of an example in which the videoconference terminal 3d transmits the resolution request information to the communication management system <NUM>, with reference to <FIG>. Other communication terminals (videoconference terminal 3a, PC <NUM>, and smartphone <NUM>) also perform the same or substantially the same operation as the videoconference terminal 3d. Accordingly, redundant description thereof is omitted below. As all the communication terminals perform the operation as illustrated in <FIG>, all the required resolutions are filled in the transmission resolution management DB <NUM> as illustrated in <FIG>.

Hereinafter, referring to <FIG>, a description is given of an image data transmission process in video calling. <FIG> is a sequence diagram illustrating an image data transmission process in video calling.

First, the communication unit 18a of the image capturing device 1a transmits image data and audio data obtained by capturing a subject or surroundings to the communication unit 38a of the videoconference terminal 3a (S101). In this case, because the image capturing device 1a is a device that is capable of obtaining two hemispherical images from which a full spherical panoramic image is generated, the image data is configured by data of the two hemispherical images as illustrated in <FIG>. The videoconference terminal 3a receives the image data and the audio data at the communication unit 38a.

Next, the data exchange unit 31a of the videoconference terminal 3a transmits, to the communication management system <NUM>, the image data and the audio data received from the image capturing device 1a (S102). This transmission includes an image data ID for identifying the image data as a transmission target. Thus, the communication management system <NUM> receives the image data and the image data ID at the data exchange unit <NUM>.

Next, the resolution controller <NUM> performs control to change the resolution of the image data received at S102, based on the required resolution stored in the transmission resolution management DB <NUM> (S103). When the resolution of the image data received at S102 is the same as the required resolution managed in the transmission resolution management DB <NUM>, the resolution controller <NUM> performs control not to change the resolution of the image data.

Next, the data exchange unit <NUM> of the communication management system <NUM> transmits, to the videoconference terminal 3d, the image data on which the control of resolution has been already performed at S103 and the audio data (S104). This transmission includes an image data ID for identifying the image data as a transmission target. Thus, the videoconference terminal 3d receives the image data, the image data ID, and the audio data at the data exchange unit 31d.

Next, the data storage/read unit 39d of the videoconference terminal 3d searches the image type management DB 3001d (<FIG>) using the image data ID received at S104 as a search key, to read out the image type information (source name) associated with the image data ID (S105). When the image type information indicates a special image (full spherical panoramic image, in this disclosure), i.e., when the image type information is "Video_Theta", the image and audio processor 33d generates a full spherical panoramic image from the image data received at S104, and further generates a predetermined-area image (S106).

Next, the data storage/read unit 39d searches the display layout management DB 3003d (<FIG>) using the image data ID received at S104 as a search key to read out the layout number associated with the image data ID, thereby determining a display area for displaying an image relating to the image data received at S104 from the plurality of display areas of the display 4d (S107). Then, the display control 34d displays the predetermined-area image in the display area determined at S107 (S108). When the image type information indicates a general image, i.e., when the image type information is "Video", the image and audio processor 33d does not generate a full spherical panoramic image from the image data received at S104. In this case, the display control 34d displays a general image.

Next, referring to <FIG>, a description is given of a state of video calling. <FIG> illustrate example states of video calling. More specifically, <FIG> illustrates a case in which the image capturing device 1a is not used, while <FIG> illustrates a case in which the image capturing device 1a is used.

First, as illustrated in <FIG>, when the camera <NUM> (<FIG>) that is built into the videoconference terminal 3a is used, that is, without using the image capturing device 1a, the videoconference terminal 3a has to be placed at the corner of a desk, so that the users A1 to A4 can be captured with the camera <NUM>, because the angle of view is horizontally <NUM> degrees and vertically <NUM> degrees. This requires the users A1 to A4 to talk while looking in the direction of the videoconference terminal 3a. Further, because the user A1 to A4 look in the direction of the videoconference terminal 3a, the display 4a has also to be placed near the videoconference terminal 3a. This requires the user A2 and the user A4, who are away from the videoconference terminal 3a, to talk in a relatively loud voice, because they are away from the microphone <NUM> (<FIG>). Further, it may be difficult for the user A2 and A4 to see contents displayed on the display 4a.

By contrast, as illustrated in <FIG>, when the image capturing device 1a is used, the videoconference terminal 3a and the display 4a can be placed relatively at the center of the desk, because the image capturing device 1a is capable of obtaining two hemispherical images from which a full spherical panoramic image is generated. This enables the users A1 to A4 to talk in a relatively small voice, because they are close to the microphone <NUM>. Further, it gets easier for the users A1 to A4 to see contents displayed on the display 4a.

Hereinafter, referring to <FIG>, a description is given of a display example on the display 4d at the site D. <FIG> are views, each illustrating a display example on the display 4a at the site D. More specifically, <FIG> is a view illustrating an example in which image data transmitted from respective ones of the image capturing device 1a (videoconference terminal 3a) and the image capturing device 1b (smartphone <NUM>) are displayed as they are, that is, without generating a full spherical panoramic image and a predetermined-area image. <FIG> is a view illustrating an example in which predetermined-area images are displayed, which are generated based on full spherical panoramic images generated from the image data transmitted from the image capturing device 1a (videoconference terminal 3a) and the image capturing device 1b (smartphone <NUM>), respectively. In this example, an image of the site A is displayed in a left-side display area (layout number "<NUM>") of the display 4d. In an upper-right display area (layout number "<NUM>"), an image of the site B is displayed. Further, in a middle-right display area (layout number "<NUM>") of the display 4d, an image of the site C is displayed. In a lower-right display area (layout number "<NUM>"), an image of the site D (own site) is displayed.

When the image data transmitted from respective ones of the image capturing device 1a (videoconference terminal 3a) and the image capturing device 1b (smartphone <NUM>), each being capable capturing a full spherical panoramic image, are displayed as they are, images are displayed as illustrated in <FIG>. In other words, the images of the site A and the site B are displayed as the front-side hemispherical image as illustrated in <FIG> and the backside hemispherical image as illustrated in <FIG>.

By contrast, when the image and audio processor 33d generates a full spherical panoramic image from the image data transmitted from respective ones of the image capturing device 1a (videoconference terminal 3a) and the image capturing device 1b (smartphone <NUM>), each being capable of obtaining two hemispherical images from which the full spherical panoramic image is generated, and further generates a predetermined-area image, the predetermined-area image, which is a planar image, is displayed as illustrated in <FIG>. Further, in both of <FIG>, the general image is displayed in the display areas of the site C and site D, because the image capturing device <NUM> and the camera <NUM> built in the videoconference terminal 3d, each being an image capturing device that obtains a general image, are used in the site C and the site D, respectively.

Furthermore, a user is able to change a predetermined area for the predetermined-area image in the same full spherical panoramic image. More specifically, when the user D1, D2 or D3 operates the operation key <NUM> or moves his/her finger on the touch panel of the display 4d, the acceptance unit 32d detects an instruction for moving the image. The display control 34d shifts, rotates, reduces, or enlarges the predetermined-area image based on the instruction detected by the acceptance unit 32d. This enables to shift the predetermined-area image so that the user A3 and the user A4 are displayed, even in a case in which the predetermined-area image displayed according to an initial setting (by default) contains only a part of the users at the site A, that is, the user A1 and the user A2 as illustrated in <FIG>.

As described heretofore, according to the present embodiment, the communication terminal such as the videoconference terminal 3a generates a full spherical panoramic image, and further generates a predetermined-area image, depending on the image type information associated with the image data ID that is transmitted along with image data. This prevents the front side hemispherical image and the back side hemispherical image from being displayed as illustrated in <FIG>.

In addition, the resolution of image data requested to the communication management system <NUM> is controlled depending on image types (S103). This prevents the resolution of an image displayed on a display from getting too low, and thereby preventing a user who views the image on the display from having difficulty in recognizing the surroundings or attendants of a communication counterpart.

Hereinafter, referring to <FIG>, a description is given of another example of the request process of the resolution. <FIG> is a sequence diagram illustrating another example of an operation of requesting the resolution.

In the above example described with reference to <FIG>, at S71, the data storage/read unit 39d reads out the image data ID and the display size from the display layout management DB 3003d. By contrast, in this example, the data storage/read unit 39d reads out the IP address in addition to the image data ID and the display size (S171). Then, the data storage/read unit 39d performs the same or substantially the same processes as the above-described steps S72 and S73 (S172, S173).

Further, in the above example described with reference to <FIG>, at S74, the data exchange unit 31d transmits the resolution request information including the image data ID and the required resolution. By contrast, in this example, the data exchange unit 31d transmits the resolution request information including the IP address that is read out at S171 in place of the image data ID (S174).

The communication management system <NUM> does not have to perform the same or substantially the same process as S75 of <FIG>, because the communication management system <NUM> receives the IP address at S174. Accordingly, after S174, the data storage/read unit <NUM> performs the same or substantially the same process as S76 (S175).

The above-described embodiments are illustrative and do not limit the present invention. Thus, numerous additional modifications and variations are possible in light of the above teachings. For example, elements and/or features of different illustrative embodiments may be combined with each other and/or substituted for each other within the scope of the present invention.

Claim 1:
A communication terminal (3d) for requesting a communication management system (<NUM>) that relays image data transmitted from another communication terminal (3a, <NUM>, <NUM>) to control resolution of the transmitted image data, the communication terminal (3d) comprising:
required resolution management means (3004d, <NUM>, <NUM>, <NUM>) for storing, for each of a plurality of image types, a display size of each of a plurality of display areas divided from an entire area displayable by a display, in association with required resolutions of image data to be displayed in the corresponding display areas, wherein
said display size corresponding to a number of pixels in the horizontal and vertical directions;
said required resolution corresponding to a number of pixels in the horizontal and vertical directions;
the image types including a planar image and a full spherical panoramic image; and the communication terminal (3d) further comprising:
receiving means (31d, <NUM>, <NUM>) for receiving, from the communication management system (<NUM>), specific image type information indicating a type of specific image data transmitted from the other communication terminal (3a, <NUM>, <NUM>);
transmitting means (31d, <NUM>, <NUM>) for transmitting, to the communication management system (<NUM>), specific required resolution, which is a resolution stored by the required resolution management means (3004d, <NUM>, <NUM>, <NUM>), for the specific image type information, in association with the size of one of the display areas in which the specific image data is to be displayed;
generating means (33d, <NUM>) for generating data of a planar image being an image of a partial area of a full spherical panoramic image based on the specific image data transmitted from the other communication terminal (3a, <NUM>, <NUM>), in a case in which the specific image type information indicates that the type of the specific image data is a full spherical panoramic image;
wherein for the same display size, the stored required resolution for a full spherical panoramic image is higher than the stored required resolution for a planar image;
display control means for controlling the display to display the generated planar image in the display area associated with the transmitted specific required resolution.