Abstract:
A method, apparatus, and gateway allows a variety of client devices to access information from current information providers, and the information is adapted to accommodate the differing formats of the client devices. Speech processing, text-to-speech (TTS), speech feature transmission on heterogeneous networks, quality of service (QoS) support for real-time applications, transcoding, transformation, publish rendering, multimedia contents analysis, and speech coding may be supported.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS  
       [0001]    This application claims the benefit of U.S. Provisional Application No. 60/294,705, filed May 31, 2001. 
     
    
     
       FIELD  
         [0002]    This invention relates generally to computers and more particularly to information retrieval using remote devices.  
         COPYRIGHT NOTICE/PERMISSION  
         [0003]    A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. The following notice applies to the software and data as described below and in the drawings hereto: Copyright© Intel, Incorporated, 2001. All Rights Reserved.  
         BACKGROUND  
         [0004]    As the Internet has evolved, users have begun to access information on remote servers using small devices, such as handheld devices, cellular phones, laptop computers, and PDAs (Personal Digital Assistants). These small client devices have smaller displays, less powerful processors, less memory, and different user interfaces than traditional laptop or desktop computers.  
           [0005]    Some of these devices suffer from problems when attempting to interface with current servers, which were designed to accommodate traditional desktop computers. First, some devices have awkward and difficult input techniques, such as selecting letters on a screen using a stylus or finger. Second, since current information providers have designed their data on the servers to have a format convenient for and compatible with traditional desktop computers, there can be a format mismatch between the data and the device attempting to receive the data and communicate it to the user. Third, the format of the data is often inconvenient for the users of small devices. Finally, small devices with a slow processor and small memory cannot easily handle the large amounts of data from current servers.  
           [0006]    Thus, there is a need for a solution that will enable a variety of devices to take advantage of the data created by information providers. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0007]    [0007]FIG. 1 depicts a block diagram of an embodiment of the invention.  
         [0008]    [0008]FIG. 2 depicts a block diagram of a stack of function blocks, according to an embodiment of the invention.  
         [0009]    [0009]FIG. 3 depicts a block diagram of an exemplary electronic device according to an embodiment of the invention.  
         [0010]    [0010]FIG. 4 depicts a flowchart of processing when the client device is a wireless or wire handset and speech is used for information retrieval.  
         [0011]    [0011]FIG. 5 depicts a flowchart of processing when the client device is a wireless or wire handset and a keyboard or pen is used for information retrieval.  
         [0012]    [0012]FIG. 6 depicts a flowchart of processing when the client device is a desktop or laptop computer and speech is used for information retrieval.  
         [0013]    [0013]FIG. 7 depicts a flowchart of processing when the client device is a desktop or laptop computer and a keyboard or pen is used for information retrieval.  
         [0014]    [0014]FIG. 8 depicts a flowchart of processing when the client device is a telephone and speech is used for information retrieval.  
     
    
     DETAILED DESCRIPTION  
       [0015]    In the following detailed description of exemplary embodiments of the invention, reference is made to the accompanying drawings (where like numbers represent like elements), which form a part hereof, and in which is shown by way of illustration specific exemplary embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, but other embodiments may be utilized and logical, mechanical, electrical, and other changes may be made without departing from the scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.  
         [0016]    In the following description, numerous specific details are set forth to provide a thorough understanding of the invention. However, it is understood that the invention may be practiced without these specific details. In other instances, well-known circuits, structures and techniques have not been shown in detail in order not to obscure the invention.  
         [0017]    [0017]FIG. 1 depicts a block diagram of an embodiment of the invention. Wireless Multimedia, a PDA, and a mobile phone  105  are shown communicatively coupled to cellular network  110 , which is communicatively coupled to gateway  115 . Gateway  115  is shown communicatively coupled to network  120 , such as the Internet or an Intranet. PDA, PC (Personal Computer), or laptop  125  is shown communicatively coupled to Network  130 , which is communicatively coupled to gateway  115 . In an embodiment, network  130  may be the Internet, but in other embodiments any suitable network may be used. Telephone  135  is shown communicatively coupled to PSTN (Public Switched Telephone Network)  140 , which is communicatively coupled to gateway  115 . Gateway  115  includes service sniffer  145 , speech coder  150 , DSR (Distributed Speech Recognition) speech recognition engine  155 , DSR portal  160 , TTS (Text To Speech) engine  165 , command interpreter engine  170 , voice portal  175 , telephony recognition engine  180 , contents search and analysis engine  183 , ACD (Adaptive Content Delivery) portal  186 , publish rendering engine  189 , transcoding engine  191 , and transformation engine  194 . In an embodiment, gateway  115  may be a functional unit that interconnects two networks with different network architectures. In an embodiment gateway  115  has the hardware configuration described below with respect to FIG. 3.  
         [0018]    [0018]FIG. 2 depicts a block diagram of stacks of function blocks, according to an embodiment of the invention. Client  205 , transmission medium  240 , gateway  250 , and network monitor  299  are shown.  
         [0019]    Client device  205  represents functions in any of client devices  105 ,  125 , and  135 , as previously described above with reference to FIG. 1. Referring again to FIG. 2, client device  205  includes speech decoder  207 , transformation decoder  209 , contents display  211 , DSR feature extractor  213 , QoS (Quality of Service) module  215 , client applications  216 , transport controller  218 , DSR feature compressor  219 , and contents ornamentor  221 . In other embodiments, client device  205  may include only selected functions.  
         [0020]    Speech decoder  207  may decompress the compressed audio from gateway  115 . Transformation decoder  209  may decode the transformed multimedia contents from gateway  115 .  
         [0021]    Contents Ornamentor  221  may ornament black-and-white multimedia contents to a higher-resolution multiple-scale gray format or higher-resolution pseudo color. Contents Ornamentor  221  also may minus adapt the multi-media contents for display on the screen size of client  205 .  
         [0022]    Contents display  211  adapts the received contents for proper display according to the physical properties of the client device. Examples of physical properties are screen size, screen resolution, and color depth. But, in other embodiments, any physical properties of the client device may be used. Contents display  211  processes the output of speech decoder  207 , transformation decoder  209 , and contents ornamentor  221  to finally display the contents at client  205 . In an embodiment, the properties of the client device may be determined using the User Agent protocol, which provides a description of client device capabilities. In another embodiment, the CC/PP (Composite Capabilities/Preference Profiles) protocol may be used, which provides a description of client device capabilities and user preferences. In other embodiments, any suitable technique for determining the capabilities of the client device may be used. When client  205  initiates a session by negotiating with gateway  115 , the type of client information (whether User Agent format, CC/PP format, or some other format) may be sent to inform gateway  115  about the client&#39;s capabilities.  
         [0023]    DSR feature compressor  219  encodes the speech feature that may be sent to gateway  115 . DSR feature extractor  213  extracts the speech feature from the user&#39;s speech. In an embodiment DSR feature extractor  213  uses MFCC (Mel-Frequency Cepstral Coefficients) to extract the speech feature. In an embodiment, the HMM (Hidden Markov Model) may be used for speech recognition. QoS Module  215  may be an interface to network monitor  299 . QoS Module  215  accepts QoS information from network monitor  299  and gives feedback information to gateway  115 . Transport Controller  218  receives QoS information from QoS Module  215  and then transfers it to gateway  115 , or vice versa. Transport controller  218  also starts transmission for heterogenous network bearers.  
         [0024]    Network Monitor  299  on the client side monitors the information stream from gateway  115  to client  205 , measures QoS parameters for this information stream, and reports this QoS information to client applications  216 .  
         [0025]    Components stack in transmission medium  240  includes DSR transmission protocols  242 , IP (Internet Protocol)  244 , and Wireless/Wireline Bearer  246 . DSR transmission protocol  242  may be an application layer protocol that may be above IP  244 . With QoS module  215  and transport controller  218 , DSR transmission protocol  242  transmits compressed speech and retrieves information efficiently and robustly. IP  244  may be a protocol that governs the breakup of data messages into packets, the routing of the packets from sender to destination, and the reassembly of the packets into the original data messages at the destination. Wireless/Wireline bearer  246  may be a transport protocol that transmits audio packets across a transmission medium. Examples of appropriate wireless/wireline bearers include GPRS (General Packet Radio Service), EDGE (Enhanced Data Rate for GSM and TDMA/136 Evolution), 3G (Third Generation Mobile Communication System), and the Internet. In other embodiments, any appropriate wireless/wireline protocol may be used.  
         [0026]    Gateway stack  250  represents a stack of functions in gateway  115 . Gateway stack  250  includes service sniffer  145 , speech coder  150 , DSR speech recognition engine  155 , DSR portal  160 , TTS engine  165 , command interpreter engine  170 , voice portal  175 , telephony recognition engine  180 , contents search and analysis engine  183 , ACD portal  186 , publish rendering engine  189 , transcoding engine  191 , and transformation engine  194 .  
         [0027]    When client  205  initiates a session by negotiating with gateway  115 , the type of client information (whether User Agent format, CC/PP format, or some other format) may be sent to inform gateway  115  about the client&#39;s capabilities. This client information may be used by ACD portal  186  to adapt the contents presentation via publish rendering engine  189 , transformation engine  194 , and transcoding Engine  191 .  
         [0028]    Publish rendering engine  189  renders a particular format, in an embodiment HTML (Hypertext Markup Language) or WML (Wireless Markup Language), for client  205 . In other embodiments, any appropriate format may be rendered. Examples of the functions of publish rendering engine  189  include converting one display page into multiple pages or converting one display line into multiple lines, including fractions of lines. But, any rendering function may be used, and the invention is not so limited.  
         [0029]    Transformation engine  194  converts a data format used in the contents retrieved from network  120  into a format supported by client device  205 . For example, some clients may not be able to display GIF (Graphics Interchange Format) images because of software or hardware constraints, so transformation engine  194  converts the GIF images into a format that the client does support, such as BMP (raster graphics stored in bitmap format), JPEG (Joint Photographic Experts Group), or black-and-white images. But, the invention is not limited to these particular formats.  
         [0030]    Transcoding Engine  191  converts the original content retrieved from network  120  within multiple fidelities and modalities and selects the proper content version for particular clients  205 . Typical examples of fidelity transcoding include summarizing, including image compression, text summarization, and video key-frame extraction. In other embodiments, any appropriate fidelity transcoding may be used. Modality transcoding may be translation, such as conversion from text to audio, or video to images. In other embodiments, any appropriate translation may be used.  
         [0031]    Contents search and analysis engine  183  receives keywords from DSR Portal  160 , voice portal  175 , or directly from service sniffer  145 . According to the given keywords, contents search and analysis engine  183  searches network  120  for appropriate contents, analyzes the contents, extracts the contents most relevant to the keywords, and then returns the results to DSR portal  160 , voice portal  175 , or directly to service sniffer  145 .  
         [0032]    TTS engine  165  translates text in the retrieved contents into audio speech. Speech coder  150  compresses audio to accommodate the bandwidth of transmission medium  240 . Command interpreter engine  170  detects and interprets the purpose for user&#39;s inputs especially for the methodology of voice inputs. Telephony recognition engine  180  performs speech recognition for voice input from client  135 . Speech recognition engine performs speech recognition for DSR portal  160 .  
         [0033]    Service sniffer  145  distinguishes between different inputs from clients and directs them to the appropriate services within gateway  115 . The different inputs from clients may be classified into three basic types: traditional telephone services (which are directed to voice portal  175 ), forthcoming DSR services (which are directed to DSR portal  160 ), and routine IP based services (which are directed to ACD portal  186 ). Service sniffer  145  acts as an interface between transmission medium  240  and gateway  115 .  
         [0034]    Voice portal  175  controls the input and output of telephony recognition engine  180 . Voice portal  175  sends speech data to telephony recognition engine  180  and retrieves the recognition result back. Either TTS engine  165  or command interpreter engine  170  processes the recognition result and either returns the recognition result to client  205  or uses the recognition result to request more services, for example by using the recognition result to search network  120 .  
         [0035]    DSR portal  160  performs analogous functions as voice portal  175 , but operates in a DSR environment and processes speech feature data.  
         [0036]    ACD portal  186  receives information retrieving commands and searches for the information in the Internet using contents search and analysis engine  183 . The results of the search are processed by transformation engine  194  or transcoding engine  191 , after which the results are returned to client  205  in the particular format supplied by publish rendering engine  189 .  
         [0037]    DSR synchronization  254  lies in the DSR receiver side and synchronizes received DSR bit-streams. DSR feature decoding  258  decodes the compressed feature data from clients  205 .  
         [0038]    QoS Daemon  256  receives QoS requesting information from client  205 , adjusts the QoS parameters of the client according to practical network conditions, and then sends these modified QoS parameters back to client  205 .  
         [0039]    Transport Controller  252  supervises QoS information of the network and other network states and then reports the state information to QoS Daemon  256 .  
         [0040]    [0040]FIG. 3 shows a block diagram illustrating an exemplary electronic device  300  according to an embodiment of the invention. The exemplary electronic device  300  includes processor  305 , storage device  310 , input device  320 , output device  327 , and network adapter  330 , all communicatively coupled via bus  315 . An unillustrated network may also be coupled to bus  315  through network adapter  330 .  
         [0041]    Processor  305  represents a central processing unit of any type of architecture, such as a CISC (Complex Instruction Set Computing), RISC (Reduced Instruction Set Computing), VLIW (Very Long Instruction Word), or a hybrid architecture, although any appropriate processor may be used. Processor  305  executes instructions and includes that portion of electronic device  300  that controls the operation of the entire electronic device. Although not depicted in FIG. 1, processor  305  typically includes a control unit that organizes data and program storage in computer memory and transfers data and other information between the various parts of the electronic device. Processor  305  receives input data from input device  320  and network adapter  330 , reads and stores code and data in storage device  310 , and optionally presents output data to a user via output device  327 . Processor  305  also sends and receives packets of information across a network using network adapter  330 .  
         [0042]    Although electronic device  300  is shown to contain only a single processor and a single system bus, the present invention applies equally to computers that have multiple processors and to computers that have multiple buses that each performs different functions in different ways.  
         [0043]    Storage device  310  represents one or more mechanisms for storing data. For example, storage device  310  may include read only memory (ROM), random access memory (RAM), magnetic disk storage mediums, optical storage mediums, flash memory devices, and/or other machine-readable mediums. Although only one storage device  310  is shown, multiple storage devices and multiple types of storage devices may be present. Further, although electronic device  300  is drawn to contain storage device  310 , the storage device may be distributed across other electronic devices. Storage device  310  contains code  340 , which has instructions capable of being executed on processor  305  to carry out the functions of the present invention. In another embodiment, the functions of the present invention may be carried out via hardware in lieu of a processor-based system. Of course, storage device  310  may also contain additional software and data (not shown), which is not necessary to understanding the invention.  
         [0044]    Bus  315  represents one or more busses (e.g., PCI (Peripheral Component Interconnect), ISA (Industry Standard Architecture), X-Bus, EISA (Extended Industry Standard Architecture), or any other appropriate bus) and bridges (also termed bus controllers).  
         [0045]    Input device  320  is that part of electronic device  300  that accepts input from a user. In an embodiment, input device  320  may be a keyboard, but in other embodiments, input device  320  may be a pointing device, mouse, trackball, keypad, touchpad, touch screen, pointing stick, microphone, or any other appropriate input device.  
         [0046]    Output device  327  communicates information to the user of electronic device  300 . Output device  327  may be a cathode-ray tube (CRT) based video display well known in the art of computer hardware. But, in other embodiments output device  327  may be replaced with a liquid crystal display (LCD) based or gas, plasma-based, flat-panel display. In other embodiments, any appropriate display device is used. In still other embodiments, a speaker that produces audio output may be used. In other embodiments, output device  327  might not be present.  
         [0047]    Network adapter  330  facilitates communication between electronic device  300  and an unillustrated network. Network adapter  330  provides a user of electronic device  300  with a means of electronically communicating information, such as packets, with a remote computer or a network logical-storage device. In addition, in another embodiment, network adapter  330  supports distributed processing, which enables electronic device  300  to share a task with other devices linked to the network. Although network adapter  330  is shown as part of electronic device  300 , in another embodiment they may be packaged separately.  
         [0048]    Electronic device  300  may be implemented using any suitable hardware and/or software, such as a personal computer available from a number of vendors. Portable computers, laptop computers, mainframe computers, handheld devices, PDAs, telephones, cellular telephones, and network computers or Internet appliances are examples of other possible configurations. Electronic device  300  is an example implementation of devices  105 ,  125 ,  135 , and/or  115 . The hardware and software depicted in FIG. 3 may vary for specific applications and may include more or fewer elements than those depicted. For example, other peripheral devices such as pointing devices, speech recognition devices, audio adapters, or chip programming devices, such as EPROM (Erasable Programmable Read-Only Memory) programming devices may be used in addition to or in place of the hardware already depicted. Thus, an embodiment of the invention may apply to any hardware configuration.  
         [0049]    As will be described in detail below, aspects of an embodiment pertain to specific apparatus and method elements implementable on computers. In another embodiment, the invention may be implemented as a computer program product for use with a computer system. The programs defining the functions of this embodiment may be delivered to a computer via a variety of signal-bearing media, which include, but are not limited to:  
         [0050]    (1) information permanently stored on non-rewriteable storage media (e.g., read only memory devices within a computer such as CD-ROM readable by a CD-ROM drive;  
         [0051]    (2) alterable information stored on rewriteable storage media (e.g., a hard disk drive or diskette); or  
         [0052]    (3) information conveyed to a computer by a communications media, such as through a computer or telephone network accessed via network adapter  330 , including wireless communications.  
         [0053]    Such signal-bearing media, when carrying processor-readable instructions that direct the functions of the present invention, represent embodiments of the present invention.  
         [0054]    [0054]FIG. 4 depicts a flowchart of processing when the client device is a wireless or wire handset and speech is used for information retrieval. Control begins at block  400 . Control then continues to block  405  where client  105  performs speech feature extraction. Control then continues to block  410  where client  105  performs feature compression. Control then continues to block  415  where client  105  transmits the compressed data to gateway  115 . Control then continues to block  420  where DSR portal  160  performs speech recognition on the compressed data. Control then continues to block  425  where command interpreter engine  170  determines the user request from the output of DSR portal  160  and sends the user request to ACD portal  186 . Control then continues to block  430  where ACD portal  186  retrieves the contents associated with the user request from network  120 . Control then continues to block  435  where ACD portal  186  adapts retrieved contents based on network bandwidth considerations and the client devices&#39;s resources. Control then continues to block  440  where ACD portal  186  sends the adapted contents to DSR portal  160 .  
         [0055]    Control then continues to block  445  where DSR portal  160  sends some of the adapted contents directly to client  105 . DSR portal  160  sends those contents directly to client  105  that link to the URL (Uniform Resource Locator) of a video, image, or audio database. A URL may be an address in a network. DSR portal  160  also sends some of the adapted contents to TTS  165 , which converts the contents to speech and then sends the speech to speech coder  150 , which compresses the speech and sends it to client  105 . DSR portal  160  chooses adapted contents to send to TTS  165  based on information in text adapted contents that indicates the contents are text intended to be published to users in audio speech format. In an embodiment, this indication may be supplied by a prompt tag, although in other embodiments any appropriate indication may be used.  
         [0056]    Control then continues to block  450  where client  105  decodes the contents and communicates it to the user. Control then continues to block  499  where the function returns.  
         [0057]    [0057]FIG. 5 depicts a flowchart of processing when the client device is a wireless or wire handset and a keyboard or pen is used for information retrieval. Control begins at block  500 . Control then continues to block  505  where client  105  gets a URL from the keyboard or pen input. Client  105  sends the URL to ACD portal  186  in gateway  115 . Control then continues to block  510  where ACD portal  186  retrieves the contents associated with the URL from network  120 . Control then continues to block  515  where ACD portal  186  adapts the retrieved contents based on network bandwidth and the resources of client device  105 . Control then continues to block  520  where ACD portal  186  sends back the adapted contents to DSR portal  160 .  
         [0058]    Control then continues to block  525  where DSR portal  160  sends some of the adapted contents directly to client  105 . DSR portal  160  also sends some of the adapted contents to TTS  165 , which converts the contents to speech and then sends the speech to speech coder  150 , which compresses the speech and sends it to client  105 . Control then continues to block  530  where client  105  decodes the encoded contents and communicates them to the user. Control then continues to block  599  where the function returns.  
         [0059]    [0059]FIG. 6 depicts a flowchart of processing when the client device is a desktop or laptop computer and speech is used for information retrieval. Control begins at block  600 . Control then continues to block  605  where client  125  performs speech-feature extraction. Control then continues to block  610  where client  125  performs feature compression. Control then continues to block  615  where client  125  transmits the compressed data to gateway  115 . Control then continues to block  620  where DSR portal  160  performs speech recognition on the compressed data. Control then continues to block  625  where command interpreter engine  170  determines the URL associated with the output of DSR portal  160  and sends the URL to ACD portal  186 . Control then continues to block  630  where ACD portal  186  retrieves the contents associated with the URL. Control then continues to block  640  where ACD portal  186  sends the contents to DSR portal  160 . Control then continues to block  645  where DSR portal  160  sends some of the contents directly to client  125 . DSR portal  160  also sends some of the contents to TTS  165 , which converts the contents to speech and then sends the speech to speech coder  150 , which compresses the speech and sends it to client  125 . Control then continues to block  650  where client  125  decodes the contents and communicates it to the user. Control then continues to block  699  where the function returns.  
         [0060]    [0060]FIG. 7 depicts a flowchart of processing when the client device is a desktop or laptop computer and a keyboard or pen is used for information retrieval. Control begins at block  700 . Control then continues to block  705  where client  125  obtains a URL from the keyboard or pen input. Client  125  sends the URL to ACD portal  186  in gateway  115 . Control then continues to block  710  where ACD portal  186  retrieves the contents associated with the URL from network  120 . Control then continues to block  720  where ACD portal  186  sends back the contents to DSR portal  160 . Control then continues to block  725  where DSR portal  160  sends some of the contents directly to client  125 . DSR portal  160  also sends some of the contents to TTS  165 , which converts the contents to speech and then sends the speech to speech coder  150 , which compresses the speech and sends it to client  125 . Control then continues to block  730  where client  125  decodes the encoded contents and communicates them to the user. Control then continues to block  799  where the function returns.  
         [0061]    [0061]FIG. 8 depicts a flowchart of processing when the client device is a telephone and speech is used for information retrieval. Control begins at block  800 . Control then continues to block  805  where client  135  sends user audio to gateway  115  via PSTN  140 . Control then continues to block  810  where voice portal  175  performs speech recognition. Control then continues to block  815  where command interpreter engine  170  determines the URL based on the output of voice portal  175 . Control then continues to block  820  where ACD portal  186  retrieves contents from network  120  based on the URL. Control then continues to block  840  where TTS engine  165  translates the retrieved contents to audio. Control then continues to block  845  where TTS engine  165  transfers the audio to client  135 . Control then continues to block  850  where client  135  plays the audio to the user. Control then continues to block  899  where the function returns.