Abstract:
An autonomous robot mainly contains an image capturing device, an interpretation device, a synthesis device, and an audio output device. The image capturing device captures pages of graphical images in which appropriate musical information is embedded, and the interpretation device deciphers and recognizes the musical information contained in the captured graphical image. The synthesis device simulates the sound effects of a type of instrument or a human singer by synthesis techniques in accordance with the recognized musical information. The audio output device turns the output of the synthesis device into human audible sounds. The graphical image of appropriate musical information is prepared in a visually recognizable form. The graphical image can also contain special symbols to give instructions to the autonomous robot such as specifying an instrument.

Description:
BACKGROUND OF THE INVENTION 
       [0001]    1. Field of the Invention 
         [0002]    The present invention generally relates to autonomous robots, and more particularly to a robotic device and a related method capable of recognizing graphical images with embedded musical information and delivering musical sounds in accordance with the musical information. 
         [0003]    2. The Prior Arts 
         [0004]    Recent researches have made significant progresses in making a robotic device to independently respond to external visual and/or audio stimulus without human involvement. Many academic and commercial prototypes have been disclosed on regular basis. To mention just a few, for example, the Sony® AIBO® is an autonomous robotic dog equipped with a camera for receiving graphical images on pictorial cards presented to it. The graphical image contains encoded instructions to trigger the robotic dog to change specific settings or to perform specific actions (e.g., dancing and singing). 
         [0005]    Other examples include the DJ robots and music playing robots from Toyota®. The DJ robot is an autonomous robot on rolling wheels that can communicate with people and behaves like it is conducting a band of music playing robots. Each of the music playing robots, either with legs or on rolling wheels, can physically play an instrument such as trumpet, tuba, trombone, and drums. The music playing robots are not really autonomous ones, but are programmed to demonstrate their agility of arms, hands and fingers. 
         [0006]    Yet another example is the Haile robot currently developed by the Georgia Institute of Technology, U.S.A. Haile is a robotic “drummer” that can listen to live players, analyze their music in real-time, and use the analytical result to play back on drums in an improvisational manner. The improvisatory algorithm enables the robot to respond to the playing of another live player. The robot can simply imitate what the other player is playing, or it can also transform its response or accompany the live player. A user can also compose music for the robot by feeding it a standard MIDI file. 
         [0007]    Despite still quite primitive, these music playing robots are found to be quite useful for educational and entertainment purposes. However, most of these robots are designed to physically operate and play a single type of instrument and, in some cases, the instrument has to be tailored for the robot&#39;s operation. On the other hand, the rhythms delivered by the robots are mostly pre-programmed in the robots or, as in the Haile robot, are learned by the robots in advance from live players. In other words, these robots cannot change what they are playing on demand, but requires some preliminary work in preparing the robots. All these, in one way or another, limit the applicability of the music playing robots. 
       SUMMARY OF THE INVENTION 
       [0008]    Accordingly, a novel autonomous robot for music playing and a related method are provided herein which combine optical recognition and sound synthesis techniques in delivering highly flexible and dynamic music performance. 
         [0009]    The autonomous robot mainly contains an image capturing device, an interpretation device, a synthesis device, and an audio output device. Usually these devices are housed in a humanoid or appropriate body. The image capturing device such as a CCD camera captures pages of graphical images in which appropriate musical information is embedded, and the interpretation device recognizes and deciphers the musical information contained in the captured graphical images. The synthesis device simulates the sound effects of at least a type of instrument or a human singer by synthesis techniques in accordance with the recognized musical information. The audio output device such as a loudspeaker turns the output of the synthesis device into human audible sounds. The audio output device is usually an integral part of the autonomous robot body, or it can be placed at a distance by appropriate signal cabling. 
         [0010]    The autonomous robot operates in a trigger-and-response manner. The graphical images of appropriate musical information such as notes on a staff or numbered notations are prepared in a visually recognizable form such as printing or writing on a board or a piece of paper. The graphical images can also contain special symbols to give instructions to the autonomous robot such as specifying a specific type of instrument. The graphical images are then presented to the image capturing device of the autonomous robot to trigger its performance as instructed by the graphical images. A series of graphical images can be sequentially presented to the autonomous robot by a human user or the autonomous robot further contains a mechanism to “flip” through the pages of graphical images, so that the autonomous robot can engage continuous music performance. 
         [0011]    A number of the autonomous robots can be grouped and perform together like a band, a chorus, a choir, or even an orchestra, by having each of the autonomous robots playing a specific role from separate sets of graphical image. For example, some may sing as tenors, sopranos, baritones, etc. Similarly, some may play violins and pianos while others play trumpets and drums. 
         [0012]    The foregoing and other objects, features, aspects and advantages of the present invention will become better understood from a careful reading of a detailed description provided herein below with appropriate reference to the accompanying drawings. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0013]      FIG. 1   a  is a schematic diagram showing the functional blocks of an autonomous robot according to an embodiment of the present invention. 
           [0014]      FIG. 1   b  is a schematic diagram showing the autonomous robot of  FIG. 1   a  interacting with a display which presents the graphical images. 
           [0015]      FIG. 1   c  is a schematic diagram showing the functional blocks of an autonomous robot according to another embodiment of the present invention. 
           [0016]      FIG. 2   a  is a schematic diagram showing a page of graphical image using numbered notation. 
           [0017]      FIG. 2   b  is a schematic diagram showing the stream of musical information contained in the page of graphical image of  FIG. 2   a.    
           [0018]      FIG. 2   c  is a schematic diagram showing the stream of musical information runs across two pages of graphical images. 
           [0019]      FIG. 2   d  is a schematic diagram showing multiple steams of musical information run across two pages with special symbols added. 
           [0020]      FIG. 2   e  is a schematic diagram showing multiple steams of musical information in a single page with special symbols added. 
           [0021]      FIG. 2   f  is a schematic diagram showing multiple steams of musical information in a single page with lyrics added. 
           [0022]      FIG. 2   g  is a schematic diagram showing multiple steams of musical information in a single page using two types of symbols to indicate continuation and to specify instrument. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0023]    According to the present invention, an autonomous robot of the present invention is basically a computing device capable of receiving visual triggers in the form of a sequence of graphical image with embedded musical information and delivering audible responses in accordance with the musical information. The autonomous robot itself is not required to have specific shape or body parts; whether it has a humanoid form or whether it has arms or legs or whether it is movable is irrelevant to the present invention. 
         [0024]    It should be noted that, even though there are quite a few prior-art robots capable of playing musical instruments (such as the Haile robot) and engaging in trigger-and-response behavior (such as the AIBO robotic dog), the present invention differs from these robots in that, in addition to using synthesis techniques for producing musical sounds of various instruments and human singers, an autonomous robot of the present invention is not pre-programmed to play a specific instrument based on some heuristic algorithm or pre-installed musical information, and the triggers (i.e., graphical images) presented to the robot is not just one-shot commands but contain time-dependent information. However, pointing out these differences is not meant to preclude the possibility that the function of the present invention is integrated with the prior art techniques in a single autonomous robot. 
         [0025]      FIG. 1  is a schematic diagram showing the internal functional blocks of an autonomous robot according to the present invention. As illustrated, the autonomous robot mainly contains at least an image capturing device  22  housed in the body  20  of the robot. A typical example of the image capturing device  22  is a CCD camera. Another typical example is a CMOS camera. A one-page-at-a-time, fax-machine-like scanning device is another possible candidate. One additional example is a handheld scanner that can scan strips of graphical images by manually moving the handheld scanner. 
         [0026]    Regardless of the technology adopted, the basic characteristic of the image capturing device  22  is that it is capable of obtaining two-dimensional graphical images from external visual triggers. For a fax-machine-like scanning device, a visual trigger is a piece of paper fed through the scanning device. For a handheld scanner, a visual trigger could be a page in a book that the scanner scans. For a camera, a visual trigger could be a frame of a display device (e.g., the panel of a LCD device, the screen of a PDA), a piece of paper, a page in a book, or writings on a white board or a pictorial card. In short, from the image capturing device&#39;s point of view, these visual triggers are all two-dimensional graphical images and these two-dimensional graphical images are presented to the autonomous robot and carried in units of “pages.” Here the term “page” is an abstraction of a frame of a display device, a piece of paper, a page in a book, or a card, as described above. 
         [0027]    Each page of graphical image contains time-dependent musical information represented by at least a stream (i.e., a linear sequence) of “notes” The “notes” can be the ordinary notes found in the music scores or numbered notations or other symbols that at least indicate the pitch and, among other information, the length of time the pitch must last and, jointly, these “notes” define a melody or rhythm.  FIG. 2   a  is an example of a page of graphical image using numbered notations to deliver the time-dependent musical information of a portion of the famous nursery song “Row, row, row your boat.” As illustrated, the graphical image may contain other special symbols to give more precise definition of the melody. For example, the underscore (“_”) and hyphen (“-”) represent the different lengths of the pitch denoted by the digits, and the dot beneath the digits lowers the pitch to a lower octave. Please note that the numbered notation shown in  FIG. 2   a  is only exemplary and there are many other possible and more sophisticated ways to deliver the time-dependent musical information, whether it is human readable or only machine-recognizable. 
         [0028]    As shown in  FIG. 1   a,  the two-dimensional graphical image captured by the image capturing device  22  is passed to an interpretation device  24  for recognition. The interpretation device  24  is the “brain” of the autonomous robot and is usually implemented as a computing device interfacing with the rest of the devices (e.g., the image capturing device  22 ) via appropriate I/O interfaces. For example, the interpretation device  24  has a conventional computer architecture with CPU, memory, buses, etc., and the image capturing device  22  (e.g., a CCD camera) is connected to the interpretation device  24  via an image capture board installed in an expansion slot of the interpretation device  24 . The most significant characteristic of the interpretation device  24  is that it is capable of performing image recognition on the graphical image delivered to it by the image capturing device  22  to extract the time-dependent musical information. Image recognition is a well-know art and many techniques have been disclosed. The subject matter of the present invention is not about the image recognition technique used and any appropriate technique can be used in the present invention. 
         [0029]    Please not that the “notes” are arranged in a pre-determined sequence, e.g., from left to right and from top to bottom on the page of graphical image if the page is held in front of the autonomous robot, as denoted by the dotted line shown in  FIG. 2   b . A very important task of the interpretation device  24  is to decipher the pre-determined sequence of “notes” so that the melody represented by the page of graphical image can be reconstructed. When multiple pages of graphical image are presented to the autonomous robot, in accordance with the sequential order of the pages presented, the melody of each page can be concatenated together into a longer melody by the interpretation device  24 , as shown in  FIG. 2   c.  The multiple pages of graphical images can be presented to the autonomous robot in various ways. In one embodiment, each page of graphical image is a pictorial card and the cards are manually shown to the image capturing device  22  one at a time by a person. In another embodiment, the pages of graphical images are pre-installed in a computer or a PDA and the pages are presented on a CRT or LCD display  10  of the computer or the PDA positioned or held in front of the capturing device  22 , as shown in  FIG. 1   b.  The presentation of the pages on the display  10  can be automatically controlled by the computer of PDA in a pre-determined speed. In yet another embodiment, an appropriate signal link is provided between the computer or PDA and the interpretation device  24 . The switch of pages therefore is controlled by the interpretation device  24  by issuing an appropriate command to the computer of PDA. This can be viewed as some kind of mechanism for “flipping” the pages of graphical image. In one additional embodiment, as shown in  FIG. 1   c,  the “flipping” mechanism  23  can be an integral part of the autonomous robot which holds pieces of paper-based pages of the graphical images and actually flips through the pages under the control of the interpretation device  24 . This automatic page flipper is already quite commonly found in advanced scanners specifically designed to automatically produce digital images of a large number of books. 
         [0030]    As shown in  FIG. 1   a,  the time-dependent musical information pieced together by the interpretation device  24  from one or more pages of graphical images is concurrently fed to a synthesis device  26  which produces synthesized sound in accordance with the musical information. The synthesized sound is then delivered via the audio output device (e.g., speaker)  28 . In one embodiment, the synthesis device  26  is able to simulate multiple types of instrument concurrently. If there is a single stream of musical information, as shown in  FIG. 2   c,  the synthesis device  26  simulates a default type of instrument. For the present embodiment, each page of the graphical image can contain multiple streams of musical information, as shown in  FIG. 2   d.  As illustrated, each page contains three streams of musical information as denoted by the dotted lines with each stream played by the synthesis device  26  simulating a particular type of instrument. To achieve this, special symbols must be positioned at predetermined locations along with the sequences of “notes.” As shown in  FIG. 2   d,  the characters “V,” “P,” and “D” precede each row of notes in a page to specify the corresponding steam of musical information to be played by simulating violin, piano, and drum. As also shown in  FIGS. 2   d  and  2   e,  the special symbols also allow the interpretation device  24  to recognize and piece together the series of rows of “notes” of the same stream, even when presented with multiple pages of graphical image. Please note that, in another embodiment, there could be multiple synthesis devices  26  with each one simulating a particular of instrument. 
         [0031]    As described above, a single autonomous robot according to the present invention is therefore able to simulate a band or an orchestra, or a group of autonomous robots of the present invention can be grouped together and, by configuring each one of them to simulate a particular instrument, play like a band or orchestra. This group of autonomous robots can have separate sets of pages of graphical images respectively, or they can all read from the same set of graphical images. The latter can be achieved by projecting the pages to a spot where each autonomous robot has its image capturing device  22  aimed at. 
         [0032]    In another embodiment where the synthesis device  26  is capable of pronouncing words using synthesized voice or pre-recoded alphabets, the autonomous robot can also be triggered to sing along with the melody. As shown in  FIG. 2   f,  which is an extension of  FIG. 2   e,  a stream of lyrics is contained in the graphical image with a special symbol “H” to signal the interpretation device  26  to simulate human voice. Please note that the words of the lyrics have to be aligned with the “notes” appropriately so that the words can be sung harmoniously. Please also note that a stream of words of the lyrics must be associated with a stream of “notes” but a stream of “notes” can be associated with multiple streams of lyrics each preceded with a special symbol for signaling the interpretation device  26  to simulate, for example, a baritone, a tenor, etc, respectively. In other words, the specification of simulating a particular type of human voice is achieved just like specifying a specific type of instrument. 
         [0033]    Another simpler way to make the autonomous robot to “sing” is to use phonetic symbols or phonograms to spell the speech sounds of the lyrics, instead of using real words. Other than this, this approach is exactly like the previous embodiment. For example, the phonetic symbols of the lyrics also have to be aligned with the “notes” appropriately so that the phonetic sounds can be produced harmoniously. With the aforementioned approaches, a single autonomous robot can sing a song, play an instrument, or do both at the same time. Additionally, a single autonomous robot or a group of autonomous robots together can sing to simulate the performance by a choir or a chorus. 
         [0034]    As shown in  FIGS. 2   d ˜ 2   f,  the special symbols are positioned in front of every row of “notes” or lyrics. However, this is not the only possibility. In another embodiment, the special symbols are replaced by two types of symbols: the continuation symbols and the instrument symbols. The continuation symbols are usually positioned in front of every row of “notes” or lyrics, as shown in  FIGS. 2   d ˜ 2   f  so that the interpretation device  26  can concatenate the series of rows of the same stream together during its image recognition process. On the other hand, the instrument symbols for specifying the simulation of a particular type of instrument can be embedded in the rows of “notes” or lyrics.  FIG. 2   g  depicts one such example with continuation symbols such as          , Δ, Ω, §, etc., and instrument symbols such as “V,” “P,” “D,” “H,” etc. An advantage of this embodiment is that, by having the instrument symbols embedded in the streams of musical information such as the “T” (for trumpet) shown in the bottommost “D” row, the autonomous robot is able to dynamically switches instruments during the delivery of a melody. For example, according to bottommost “D” row in  FIG. 2   g,  the autonomous robot will initially simulate, among other types of instruments and human voices, the instrument drum and then subsequently switch to simulate the instrument trumpet. 
         [0035]    As shown in  FIG. 1a , the output of the synthesis device  26  is fed to, converted into analog signals, and presented as human audible sounds to the surroundings by the audio output device  28 . A typical audio output device  28  contains one or more loudspeakers driven by an appropriate amplification circuit. The audio output device  28  can be completely housed inside the body  20  of the autonomous robot or, in some embodiments, the loudspeaker or loudspeakers are placed at a distance from the body  20  and connected to the amplification circuit inside the body  20  by appropriate wired or wireless connection. 
         [0036]    Although the present invention has been described with reference to the preferred embodiments, it will be understood that the invention is not limited to the details described thereof. Various substitutions and modifications have been suggested in the foregoing description, and others will occur to those of ordinary skill in the art. Therefore, all such substitutions and modifications are intended to be embraced within the scope of the invention as defined in the appended claims.