Patent Publication Number: US-6714238-B2

Title: Video/audio communication system with confirmation capability

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates to a video/audio communication, a video communication system, a transmission terminal and a reception terminal, and more particularly to a video/audio communication, a video communication system, a transmission terminal and a reception terminal for mutual audio and/or video communication among plural terminals present in scattered locations on a network. 
     2. Related Background Art 
     There is recently developed a video/audio communication system utilizing a computer network and allowing to recognize the status of one or plural distant sites in one site or in mutual manner (for example so-called awareness system). Such video/audio communication system is ordinarily composed by connecting plural computers or communication terminals, each connected to a video camera of which image taking direction and/or image taking magnification is externally controllable and a microphone which is non-directional or has controllable directionality, to a computer network, in such a manner that the image and the voice respectively entered by such camera and microphone can be transmitted to the network. 
     Also a terminal receiving such image and voice from the network selectively displays and outputs the plural images and voices from plural sites respectively on an image display device and from a speaker. 
     However, in such conventional video/audio communication system, it has been difficult to identify, in the image transmitting side, by whom the image taken by its camera is observed (by which terminal the image is received and displayed). Such drawback is not limited to the camera but is present also in other resources such as the storage device. 
     Also in the remote controlling side of the camera, the obtained image lacks the realistic feeling because the input and output status of the voice transmitted with the image does not change at all even when the image taking direction of the camera (by panning and tilting) and the image taking magnification thereof (by zooming) are varied. 
     An object of the present invention is to resolve the above-mentioned drawbacks individually or entirely and to provide a video communication system allowing the image transmitting side to easily confirm the operator of the camera or the operation status thereof. 
     Another object of the present invention is to provide a video/audio communication system capable of improving the realistic feeling by varying the voice input and output in linkage with the camera operation. 
     The above-mentioned objects can be attained, according to a preferred embodiment of the present invention, by a video/audio communication system or a video communication system for video/audio or video communication through a network, comprising information providing means in the image transmitting side, for giving information to the observed person, wherein the information displayed on the information providing means is varied according to the zooming operation of the image input means, whereby the status of operation of the image input means can be easily recognized at the image transmitting side. 
     Also in such system, the observer of the transmitted image and the operator of the image input means can be readily recognized by the display of at least either of the still icons and the names of such observer and the operator. 
     Also the status of the zooming operation can be readily recognized by varying the magnification of the display on the information providing means in correspondence with the zooming magnification of the image input means. 
     SUMMARY OF THE INVENTION 
     An object of the present invention is to resolve the above-mentioned drawbacks individually or entirely. 
     Another object of the present invention is to facilitate understanding by the image transmitting side, how the resources of its terminal are utilized. 
     Still another object of the present invention is to facilitate understanding by the image transmitting side, by whom the image taking device of its terminal is operated. 
     The above-mentioned objects can be attained, according to a preferred embodiment of the present invention, by a video/audio communication system or a video communication system for video/audio or video communication through a network, comprising information providing means in the image transmitting side, for giving information to the observed person, wherein the information displayed on the information providing means is varied according to the zooming operation of the image input means, whereby the status of operation of the image input means can be easily recognized at the image transmitting side. 
     Also there is provided a system, in which the observer of the transmitted image and the operator of the image input means can be readily recognized by the display of at least either of the still icons and the names of such observer and the operator. 
     Still another object of the present invention is to provide a system terminal having novel functions. 
     Still other objects of the present invention, and the features thereof, will become fully apparent from the following description of the embodiments, which is to be taken in conjunction with the attached drawings. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     FIG. 1 is a schematic block diagram of an embodiment of the present invention; 
     FIG. 2 is a view showing an example of the image display and the camera operation in the above-mentioned embodiment; 
     FIG. 3 is a view showing an example of the image displayed on a monitor  46 ; 
     FIG. 4 is a view showing an example of the magnified display of an icon  66  on the monitor  46 ; 
     FIG. 5 is a view showing the process configuration of the present embodiment; 
     FIG. 6 is a view showing an example of the management table managed by the access management process of the present embodiment; and 
     FIG. 7 is a view showing an example of the camera control table managed by a camera control server  74 . 
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     Now the present invention will be clarified in detail by a preferred embodiment thereof, with reference to the attached drawings. 
     FIG. 1 is a schematic block diagram of an embodiment of the present invention, wherein video/audio communication terminals  10 - 1 ,  10 - 2 ,  10 - 3 ,  10 - 4 , . . . connected to a network  12  for audio and video transmission and reception have an identical configuration. More specifically, each of the video/audio communication terminals  10 - 1 ,  10 - 2 ,  10 - 3 ,  10 - 4 , . . . is provided, as in the ordinary computer, with a CPU  20  for entire control, a main storage  22 , a secondary storage  24  (for example a hard disk device), a bit map display  26 , a mouse  28  serving as a pointing device and a network interface  30 . 
     Each of the video/audio communication terminals  10 - 1 ,  10 - 2 ,  10 - 3 ,  10 - 4 , . . . is further provided, as the video input/output means, with a video camera  32  of which image taking direction, image taking magnification, focus state and power on/off state are arbitrarily controllable, a camera control device  34  for directly controlling at least one of the panning, tilting, zooming and focusing of the video camera  32 , and a video capture device  36  for capturing the output image signal of the video camera  32 , and, as the audio input/output means, with a microphone  38  of which directionality (spreading, direction and sensitivity) is controllable by an external control signal, an audio input control device  40  for controlling the directionality and the input level of the microphone  38  according to an instruction from the CPU  20 , a speaker  42  of which sound field (spreading, direction and depth) is controllable, and an audio output control device  44  for controlling the sound field of the speaker  42  according to an instruction from the CPU  20 . Such sound field control can be achieved by an already known technology. 
     Each of the video/audio communication terminals  10 - 1 ,  10 - 2 ,  10 - 3 ,  10 - 4 , . . . is further provided with a monitor  46  mounted on the video camera  32 , and a video display control circuit  48  for causing the monitor  46  to display the image obtained by the video camera  32 , according to the operation status thereof. 
     The speaker  42  may be composed either of a single speaker with controllable sound field or of a speaker device consisting of plural speakers so combined as to render the sound field controllable. Also the monitor  46  may be a display window in the image area on the bit map display  26 . 
     The video output of the video camera  32  is subjected to A/D conversion and compression in the video capture device  36  and is transmitted through the network I/F  30  to the network  12 . For simultaneous delivery of the image to plural sites, there is employed multi-cast communication or broadcast communication. Also the audio input to the microphone  38  is subjected to A/D conversion and compression by the audio input device  40  and is transmitted through the network I/F  30  to the network  12 . As in the case of image, the audio signal is transmitted by the multicast communication or the broadcast communication for simultaneous delivery to plural sites. 
     As will be explained later, the audio input control device  40  controls the directionality (spreading, direction and sensitivity) of the microphone  38 , so as to match the video input state of the video camera  32 , corresponding to the panning, tilting and zooming control of the video camera  32  by the camera control device  34 . 
     The audio output control device  44  expands the input audio information and effects D/A conversion for supply to the speaker  42 . In this operation, the audio output control device  44  controls the spreading, direction and depth of the output sound field of the speaker  42 , so as to match the input status of the video camera connected to the communication terminal at the source of the audio information, according to the operation status of such video camera. 
     The camera control device  34 , the audio input control device  40  and the audio output control device  44  can also be controlled from a remote site through the network  12 . The format and the compression method for the video and audio data are not particularly limited. In the present embodiment, the video/audio communication terminals  10 - 1 ,  10 - 2 ,  10 - 3 ,  10 - 4 , . . . are composed of ordinary work stations. 
     FIG. 2 in conjunction with FIG. 1 show an example of the image field displayed on the bit map display  26 , on which operated is a window display system capable of displaying plural windows. 
     A symbol  50  indicates that the video transmission is in progress, and, while this symbol  50  is displayed on the image field, the image taken by the camera  32  is transmitted to the network  12 . A video transmission process to be explained later displays this symbol  50  on the image field. A symbol  52  indicates that the audio transmission is in progress, and, while this symbol  52  is displayed on the image field, the audio input by the microphone  38  and the transmission of the input audio signal to the network  12  are rendered effective. A symbol  54  indicates that the audio reception is in progress, and, while this symbol  54  is displayed on the image field, the audio output by the speaker  42  from the network  12  is rendered effective. 
     These symbols are naturally not limited to those illustrated but can be of any designs displayable on the image field. 
     A scattered site video display window  58  displays selected ones among the images delivered from other communication terminals (for example, in case of the bit map display  26  of the terminal  10 - 1 , the images from the terminals  10 - 2 ,  10 - 3 ,  10 - 4 , . . . ). The images to be displayed are selected for example by a menu therefor, but such selecting method is not critical. The image from another terminal is renewed at a rate matching the purpose, such as a moving image or a still image renewed for example at every 10 seconds. The transmission rate and the renewal rate are naturally limited by the transmission capacity of the network  12  and the processing capacity of the communication terminals  10 - 1 ,  10 - 2 ,  10 - 3 ,  10 - 4 , 
     A camera control panel  60  is provided with direction buttons  60   a  for panning (lateral) operation and tilting (vertical) operation, a zoom button  60   b  for zooming operation, a host name display area  60   c  for entering and displaying the host name of the video/audio communication terminal to which the camera to be controlled is connected, an access button  60   d  for requesting the control right for the camera connected to the video/audio communication terminal displayed in the host name display are  60   c , and a release button  60   e  for abandoning such control right. These buttons  60   a ,  60   b ,  60   c ,  60   d  can be operated in the known method with the mouse  28 . 
     The monitor  46  is composed of small image display means, such as a liquid crystal display panel, fixed to a side of the video camera  32 , and changes the direction according to the panning and/or tilting of the video camera  32 . The monitor  46  displays the information of the camera operator controlling the video camera  32  and of the receiver receiving the image of the video camera  32 . It may also display the control status of the camera. 
     FIG. 3 in conjunction with FIG. 1 show an example of the image field displayed on the monitor  46 , including a list  62  of the users receiving (observing) the image from the video camera  32 , a name  64  of the user who controls the video camera  32 , and an icon (still image)  66  representing the face of the user who controls the camera. Icons representing the faces of the users are stored in advance as an icon file in the secondary storage  24 , and a required icon is displayed in the position shown in FIG.  3 . The icon  66  is displayed with a magnification corresponding to the zoom ratio of the operated camera  32 . For example, if the camera  32  is in a zoomed-in state, the icon  66  is displayed in a magnified manner as shown in FIG.  4 . It is therefore made evident that the camera  32  is in a zoomed-in state. As the zoom-in of the camera corresponds to the approaching of the communication partner, such magnified display of the image of the partner provides a natural feeling. 
     The monitor  46  may also display other information than the user name and the icon, and may also be composed of a CRT or a display device of any other type, instead of the liquid crystal display. Furthermore it may be composed of an external display unit. 
     In the following there will be explained the control of the directionality of the microphone  38  and the sound field of the speaker  42 , corresponding to the panning, tilting or zooming operation of the video camera  32 . A panning or tilting operation of the video camera  32  varies the image taking direction thereof, and a zooming operation varies the image taking area (image angle) thereof. In a video or television conference, as the operator of the terminal listens to the voice from the partner and talks to the partner while observing the received and displayed image, the realistic feeling can be increased by controlling the voice according to the manipulation of the image from the partner terminal. Such control is not limited to the volume only but may also include the control of the sound field. In the present embodiment, the input characteristics of the microphone  38  and the output characteristics of the speaker  42  are controlled in linkage with the operation of the camera. 
     For example, when the direction of the video camera  32  is varied by the panning and/or tilting operation, the directions of the microphone  38  and the speaker  42  are varied according to the direction of the camera  32 . With such change in the direction of the microphone  38 , the voice in the image taking direction of the camera  32  can be picked up and transmitted to the camera operator. Also the change in the direction of the speaker  42  varies the output direction of the voice from the camera operator, thus allowing the user of the terminal of the camera  32  to realize the positional change of the camera operator. 
     In case the image taking range of the video camera  32  is made narrower by a zooming operation, the directionality of the microphone  38  and the spreading of the sound field of the speaker  42  are both made narrower, according to the image taking range of the video camera  32 . Since the attention is concentrated on a specified object in such case, the directionality of the microphone  38  and the sound field of the speaker  42  are so controlled as to preferentially input and output the voice of such object. In addition, the sensitivity of the microphone  38  is elevated, and the depth of the sound field of the speaker  42  is made larger. On the other hand, in case the image taking range of the video camera  32  is made wider by a zooming operation, the spreading, direction and sensitivity of the microphone  38  and the spreading, direction and depth of the sound field of the speaker  42  are controlled in the opposite direction. 
     As explained above, the audio input/output characteristics (spreading, direction and sensitivity of the directionality of the microphone  38  and spreading, direction and depth of the sound field of the speaker  42 ) are matched with the video taking operation status (panning, tilting and zooming). 
     FIG. 5 in conjunction with FIG. 1 show process configuration or software configuration of the present embodiment. A video transmission process  70  executes video capture and delivery and controls the display of the symbol  50 . A scattered site image display process  72  displays the images of 6 sites arbitrarily designated among the images delivered from other plural sites, in the areas of the scattered site image display window  58 . A camera control server  74  controls the camera  32  by receiving a camera control command through the network  12  and transferring such command to the camera control device  34 . The camera control server  74  also sends control commands to the audio input control device  40 , the audio output device  44  and the video display control device  48  according to the camera control, thereby controlling the input characteristics of the microphone  38  and the output characteristics of the speaker  42 , and varying the image displayed on the monitor  46 . 
     A camera control client  76  displays the camera control panel  60  on the image field of the display  26 , and transmits a camera control command from such control panel, through the network  12 , to the camera control server  74  of a video/audio communication terminal displayed in the host name display area  60   c.    
     An audio transmission process  78  executes audio capture and delivery and controls the display of the symbol  52 . An audio reception process  80  executes simultaneous or selective audio output by mixing the audio signals transmitted from the scattered sites, and controls the display of the symbol  54 . 
     The video transmission process  70 , the scattered site video display process  72 , the camera control server  74 , the camera control client  76 , the audio transmission process  78  and the audio reception process  80  function on each of the video/audio communication terminals  10 - 1 ,  10 - 2 ,  10 - 3 ,  10 - 4 , 
     An access management process  82  manages the video/audio delivery and the remote control of all the video/audio communication terminals  10 - 1 ,  10 - 2 ,  10 - 3 ,  10 - 4 , . . . connected to the network  12 , and functions in only one of such terminals connected to the network  12 . 
     FIG. 6 in conjunction with FIG. 1 show the management table managed by the access management process  82 , wherein a transmission process list  84  has a field  84 - 1  containing the names of all the video/audio communication terminals which are connected to the network  12  and for which the video transmission process  70  is operative, namely the names of the communication terminals delivering the video information (in case of the present embodiment, host names of the work stations employed), and a field  84 - 2  of the pointers to the scattered site video display list  86 . 
     The scattered site video display list  86  is provided corresponding to the entries of the transmission process list  84 , and registers the names and the log-in names of all the video/audio communication terminal receiving the image delivered from the corresponding transmission list. As an example, in case of FIG. 6, the image delivered from the host  3  is received (observed) by a person  1  (log-in name) of the host A (reception host), a person  2  (login name) of the host B (reception host), a person  3  (log-in name) of the host C etc. The log-in name is the name used for logging in the work station. 
     The reception process list is renewed when the scattered site video display process  72  is activated and when the camera images to be displayed are switched in the scattered site video display process  72  for example by the menu. In such operation, data including a log-in name/activating host name which is activating the scattered site video display process  72  and a new list of the host names of the cameras to be displayed are supplied, as a renewal request, from the scattered site video display process  72  to the management process  82 . 
     FIG. 7 in conjunction with FIG. 1 show an example of the management table managed by the camera control server  74 . A camera parameter management table  88  is renewed at each reception of the camera control command by the camera control server  74  through the network. Based on the camera parameter management table, the camera control server  74  sends control commands to the camera control device  34 , the audio input control device  40 , the audio output device  44  and the video display control device  48 , thereby controlling the camera, the input characteristics of the microphone  38 , the output characteristics of the speaker  42  and the display size of the icons on the monitor  46 . 
     A content corresponding to a list  90  of the camera image observers and the camera operator is displayed on the monitor  46 , by the video display control device  48 . The icon of the camera operator is read from the secondary storage, based on the icon file name in the list  90 , and is displayed. The list  90  is renewed, based on the renewal request transmitted from the access management process  82 . Such renewal request is generated in the following situation. At first an information renewal request for the reception process list is issued from the management process  82  to the camera control server  74 , which in response renews the list  90 . 
     With respect to the camera operator, the information renewal request is issued to the access management process  82  when the host name to be controlled is entered in the camera control panel  60  displayed by the camera control client  76  and the access is achieved by the actuation of the access button  60   d , or when the access is released by the actuation of the release button  60   e . In response the access management process  82  similarly sends an information renewal request to the corresponding camera control server  74 , which in response renews the list  90 . The present invention is not limited by the kind of the network to be employed in the transmission of the above-mentioned information. For example there can be employed an Internet or a network based on IEEE 1394. 
     As explained in the foregoing, at the image transmitting side, the camera capable of panning, tilting and zooming operation is associated, at the side thereof, with a monitor  46  which is so mounted as to change the direction according to the image taking direction of the camera and, on such monitor, there is displayed information including the still icons or names of the observers of the image taken by the camera and the operator controlling the camera, and such information displayed on the monitor  46  is magnified or reduced in size according to the zooming ratio of the camera, whereby the user observed by such camera can more easily recognize the zoom ratio of the camera and the access information of the observers. Also the directionality (spreading, direction and sensitivity) of the microphone  38  and the sound field (spreading, direction and depth) of the speaker  42  are controlled according to the panning and/or tilting angle of the camera and the zoom ratio thereof, whereby the audio information transmitted together with the video information can be varied matching the change in the image by the camera control, thereby improving the realistic feeling to the observers. 
     The present invention has been explained by an embodiment as a television conference system which deals with moving images, but it is also applicable to a system dealing with still images. In such case, the image input means can be an original document scanner or other means such as a still video camera. 
     The present invention is applicable to the data processing not only in a system consisting of plural equipment as shown in FIG. 1 but also in an apparatus consisting of a single equipment. 
     Also the objects of the present invention can be naturally attained by supplying a system or an apparatus with a memory medium storing program codes of a software which realizes the functions of hosts and terminals in the foregoing embodiment and reading and executing such program codes, stored in the memory medium, by a computer (or a CPU or a MPU) of such system or apparatus. 
     In such case, the program codes themselves read from the memory medium realize the function of the foregoing embodiment, and the memory medium storing such program codes constitutes the present invention. 
     The memory medium for supplying the program codes includes, for example, a floppy disk, a hard disk, an optical disk, a magnetooptical disk, a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory card and a ROM. 
     Also there are included not only the case where the functions of the foregoing embodiment are attained by the execution by the computer of the read program codes but also the case where an operation system functioning on the computer executes all or a part of the actual process based on the instructions of such program codes and the functions of the foregoing embodiment are attained by such process. 
     Furthermore there is included a case where the program codes read from the memory medium are stored in a memory provided in an expansion board connected to or inserted in the computer and a CPU or the like provided on such expansion board executes all or a part of the actual process based on the instructions of such program codes whereby the functions of the foregoing embodiment are attained by such process. 
     As will be readily understandable from the foregoing description, the foregoing embodiment allows the image transmitting side to easily and constantly confirm, in visual manner, the receivers of the transmitted image and the operator controlling the image input means of such image transmitting side. 
     Also the image of the camera operator, the directionality of the audio input and the sound field of the audio output are varied according to the panning, tilting and zooming operations of the camera, so that the video information and the audio information can be matched in the feeling of the observer, thus providing a system with improved realistic feeling.