Abstract:
There is provided a multi-modal mobile platform system for permitting a user to interact and exchange data through a mobile device. It will be appreciated that the present invention may be employed on any mobile device such as a cellular phone or Personal Digital Assistant (PDA) and the like. It is contemplated that the multi-modal platform (MMP) provides the basic constructs for a unified multi-modal data access and query platform that will overcome the deficiencies of the services based on a uni-modal platform such as web-only services or text messaging only services. The present system may use a Voice Portal, a Text Messages Portal, an eMail Portal, a Web Portal, a Downloadable Application Portal, or a WAP or Mobile Web Portal to facilitate communication between a user and third parties.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application claims the benefit of U.S. Provisional Application No. 60/979,194, filed on Oct. 11, 2007, the teachings of which are expressly incorporated by reference. 
     
    
     STATEMENT RE: FEDERALLY SPONSORED RESEARCH/DEVELOPMENT 
       [0002]    Not Applicable 
       BACKGROUND 
       [0003]    1. Field of the Invention 
         [0004]    This invention relates generally to the field of data communication on mobile devices and more particularly, to multi-modal platform data querying for mobile devices using various portals for communication and accessing data. 
         [0005]    2. Related Art 
         [0006]    Information networks, such as the Internet, have blossomed from a medium for simple data exchange and messaging to the fastest growing, most innovative medium for information exchange and commerce. Virtual shopping malls, buying services, and other types of sophisticated information exchanges are being employed by an ever increasing number of retail merchants, commercial enterprises, and consumers alike. With this reallocation of commerce and business being conducted over the Internet, retailers and service providers expend vast resources to provide visually appealing and technologically efficient communication exchanges for users to access data. 
         [0007]    With the rapid rise in popularity of Internet commerce and information services, and the rapid evolution of computer and communications technologies, great strides have been made in improving the timeliness, quality, quantity, and, perhaps most importantly, the types of communication platforms for exchanging data or communicating. Whereas in the past, the data was primarily exchanged through file transfer and with the exchange of text-based messages, today data networks are routinely used for distributing elaborate, interactive, real-time graphical displays, real-time audio, and real-time video. These technological improvements greatly increase user appeal to adapt to such communication methods and resultantly generate increased traffic and greater revenues. The mobile device industry has embraced data communication from the advent of the Personal Digital Assistant (PDA) to the ever dependent reliance placed on the cellular phone. 
         [0008]    Prior art content providers often utilize an uni-modal approach for data communication on mobile devices. Specifically, if a user were to query the content provider by using one communication method, the content provider would return the query results via the same communication method. One such example is with Google Text. Google Text permits users to submit a query for a business address via sending an SMS text message. Subsequently, Google will return the results using the same communication method or a return SMS text message. 
         [0009]    However, such static and rigid communication rules are not always user friendly or the most an efficient means to foster communication. This is particularly true as various industries begin to adopt business models or strategies to incorporate mobile devices. For example, SMS text messages may be useful to submit a query for a business address, but it may not be the ideal communication method for the return query results especially if the results may include a large amount of data that would require multiple SMS text messages or even if the user cares to access the data when her attention should be focused, such as while driving a car. Alternatively, a user may submit a query via a text message desiring to access the query results upon a different computing device such as a personal computer. 
         [0010]    Accordingly, there is currently a need in the art for an improved system for facilitating data communication. 
       BRIEF SUMMARY 
       [0011]    In accordance with the present invention, there is provided a multi-modal mobile platform system for permitting a user to interact and exchange data through a mobile device. It will be appreciated that the present invention may be employed on any mobile device such as a cellular phone or Personal Digital Assistant (PDA) and the like. It is contemplated that the multi-modal platform (MMP) provides the basic constructs for a unified multi-modal data access and query platform that will overcome the deficiencies of the services based on a uni-modal platform such as web-only services or text messaging only services. 
         [0012]    A basic embodiment of the present system may include a three tiered architecture having a User Access Portal Layer, an Application Layer, and a Database and Data Access Layer. The User Access Portal Layer may provide multiple portals to facilitate user communication. In this regard, a portal allows a user to communicate with or submit a query to a content provider or a data source. In a preferred embodiment of the present invention, the MMP may include six portals for user communication: Voice Portal, SMS Portal, Mobile Web Portal, eMail Portal, Online Web Portal, and Downloadable Application Portal. However, it is contemplated that any communication modality that transmits data may be employed with various aspects of the present system. 
         [0013]    The Application Layer is where the business logic may be programmed and implemented and consequently generates query results for the system. The Database and Data Access Layer is the internal data, customer information, customization information, advertising messages and business data may be deposited and stored. It is contemplated that any communications modality such as the Internet, wireless technology, proprietary network and the like may be employed to foster communication between the various layers of the present system. 
         [0014]    The MMP advantageously provides a unified experience for the user for each respective portal. In other words, regardless of the portal used to query the data, the same results may be returned with appropriation for the capability of the portal. Therefore, while the presentation of the data is changed to suit the portal used, the core information is retained to ensure the user receives the desired data. In addition, the interaction history for the user is unified and consolidated, and may further be available for user inspection in one normalized format regardless of the portal used to make the query. For example, people searching for information via voice or text message may find their search history easily on the web using their personal computer (PC). 
         [0015]    Another object of the present invention is to provide a platform that allows users to cross technologies and thereby use the best available mode of access for initial query and the subsequent response. For example, a user that queries the data using voice through the Voice Portal, and can request to receive the data response in text message format through the Text Message Portal so the user can use the persistence of the SMS text data in their mobile device as oppose to the volatility of the voice response on their phone. For example, a user who queries a Real Estate data source through the voice portal, can chose to receive the price and other characteristics of the property as a text message through SMS text which will give the data more persistence on user&#39;s mobile phone. 
         [0016]    Another feature of the MMP is to provide advertisers with an efficient and target vehicle for providing content laden messages such as commercial, informational, educational messages to users in a variety of formats. The messages may be directed to users depending on the mode of access or user preference. A text message user may receive text message advertising or informational message whereas a voice portal user may receive the same informational message in voice format. At the same time, the user may specify their preference in format for receiving messages, either during current or a previous interaction with the platform, or on the web portal when they log onto the web site associated with the platform. For example, even though the user is using a phone call and therefore the voice portal, the advertisement can be sent as a text message. Similar examples may be made for other portals. It is contemplated that content providers may also determine which portal data may be exchanged with users. 
         [0017]    It is further contemplated that the MMP may provide a platform for advertisers to direct advertisements to interested audiences by allowing advertisers to input data such as user demographics, user customization, and the user query itself as input criteria for finding an appropriate advertisement for a particular user. For example, a user that may use the Voice Portal to search for the prices of digital cameras may be presented with search results and related advertisements. Specifically, the user may be presented with a voice message that provides promotional information about digital camera batteries, which consequently is a relevant and targeted message in the same format that user has opted to use. It is contemplated that any data may be used by advertisers to target users. 
         [0018]    The MMP may also provide advertisers with cross-platform correlated data. Particularly, the MMP may provide a relevant advertisement to the user when user opts to use a different portal than the one used for the initial query. In the digital camera example provided above, should the user employ the mobile web portal to look up the history of queries, the user may be presented with a relevant promotional message about the Digital Camera memory on the mobile web portal even though the initial query resulting in this search record was done using the Voice portal. It will be appreciated that such cross correlation of portal interactions and common backend functions are amongst the most power features of the MMP. 
         [0019]    Another object of the present invention is to provide relevant advertising messages to the user based on advertisements that the user has previously received from other portals. Specifically, the advertisement history from various portals may be made available as input criteria to the advertising subsystem to choose the best targeted ad in the current interaction and current portal. As an example, a brand advertiser such as Nikon may decide that if the user hears a Nikon ad on the Voice Portal when they are searching for a digital camera, the advertiser may want the user to see a Nikon ad on the Web Portal the next time the user visits the web site in order to increase relevant impressions on the user through association and audio/visual memory. It is contemplated that any portal may be associated with such targeted advertising. 
         [0020]    It will be appreciated that the MMP is not a search engine and does not perform complex search operations. In contrast, it provides an application level gateway for various pieces of data that are provided by other software packages such as a search engine or a comparison shopping site, a calendar service, a blog service, or the like. Additionally, it is advantageous that the MMP provides access to the same data via various mobile portals. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0021]    These and other features and advantages of the various embodiments disclosed herein will be better understood with respect to the following description and drawings, in which like numbers refer to like parts throughout, and in which: 
           [0022]      FIG. 1  is a block diagram illustrating multi-modal mobile plateform and various communication portals interacting with online product and sales data, and with online mobile sales and commerce in accordance with various aspects of the present invention; and 
           [0023]      FIG. 2  is a block diagram illustrating the overall architecture of the MMP including the relationships between the User Access Portal Layer, Application Layer, and Database and Data Access Layer. 
       
    
    
       [0024]    Common reference numerals are used throughout the drawings and the detailed description to indicate the same elements 
       DETAILED DESCRIPTION 
       [0025]    The detailed description set forth below is intended as a description of the presently preferred embodiment of the invention, and is not intended to represent the only form in which the present invention may be constructed or utilized. The description sets forth the functions and sequences of steps for constructing and operating the invention. It is to be understood, however, that the same or equivalent functions and sequences may be accomplished by different embodiments and that they are also intended to be encompassed within the scope of the invention. 
         [0026]    In accordance with one aspect of the present invention, there is provided a Multi-modal Mobile Platform system (MMP) to provide multi-modal access for data communication.  FIGS. 1 and 2  illustrate the MMP  10  in accordance with various aspects of the present invention. In the present embodiment, the MMP  10  has a three-tiered architecture including a User Access Portal Layer  12 , an Application Layer  14 , and a Database and Data Access Layer  16 . However, it is contemplated that the MMP  10  may be designed having a single tiered architecture or multi-tiered architecture and that the functional scope of the present system may be incorporated throughout each layer individually or collectively. 
         [0027]    The User Access Portal Layer  12  provides users with multi-modal access to the MMP and consists of at least one portal for facilitating data communication. A portal is a data exchange modality that allows a user to communicate with a data source or a content provider and the like. In the present embodiment, the User Access Portal Layer comprises six portals including a Voice Portal  18 , SMS Portal  20 , WAP or Mobile Web Portal  22 , Web Portal  24 , an eMail Portal  26 , Downloadable Application Portal  28 , and an Advertisers Portal  30 . However, a person having ordinary skill in the art will understand that the present system may employ any number of data communication portals. 
         [0028]    The Voice Portal (VP)  18  is used to communicate with the user through conventional voice technology. In this regard, the user dials a telephone number from standard PSTN  32  (Public Switched Telephone Network) or a mobile telephone network and connects to the portal. The portal subsequently identifies the user using an authentication modality such as the ANI (Automatic Number Identification) of the user&#39;s telephone and proceeds to provide the authorized services to the user by connecting the user to the appropriate application from the Application Layer  14 . The applications served to the users from Application Layer  14  are voice enabled applications and may use VXML (Voice XML) for implementation of their logic. However, it is contemplated that any programming language may be employed to provide the requisite logic for this application. 
         [0029]    The SMS Portal (SMSP)  20  is used to communicate with the user through SMS protocol or Text Messaging. The user sends a query addressed to the data as a text message to a pre-assigned short code SMS number. The SMSP  20  receives the text message and then authenticates the user by verifying the user&#39;s identity. It is contemplated that any authentication technique may be employed to verify the user&#39;s identity, including using the incoming telephone number of the text message. Subsequently, the SMSP  20  proceeds to provide the authorized services to the user by connecting the user to the appropriate application from the Application Layer  14 . The SMSP  20  then re-appropriates the result and returns it to user in the form of a text message. 
         [0030]    The WAP or Mobile Web Portal (WAPP)  22  is used to communicate with the MMP  10  through a mobile device that is capable of accessing the Internet  34  using a browser. It is contemplated that the user points their mobile device browser to the MMP&#39;s URL (Universal Resource Locator). It is contemplated that the mobile device may access the URL through any telecommunications connectivity modality such as, but not limited to, WAP (Wireless Application Protocol), GPRS (General Packet Radio Service), EDGE (Enhanced Data for GSM), 3G (3 rd  Generation Data), 4G (4 th  Generation Data), WiFi, or WiMAX or the like. It is further contemplated that the WAPP  22  may provide output results or query results in a programming language such as WML (Wireless Markup Language), XHTML, HTM/JavaScript/Flash, and the like depending on the compatibility of the mobile device&#39;s browser. The mobile device&#39;s browser may be determined based on the UA (User Agent) information obtained from the browser. The WAPP  22  subsequently asks the user to login and upon authentication connects the user to the appropriate application from the Application Layer  14 . The portal then re-appropriates the result and returns it to user in the form appropriate for the small screen WAP or Mobile Web enabled devices. 
         [0031]    The Web Portal (WP)  24  is the online web portal for users to access data sources available to the MMP through the Database and Data Access Layer  16 . The WP  24  may also be used to search through records of existing interactions with the MMP  10  which could have been produced through the use of any of the portals. In addition the WP  24  allows the user to configure and customize the behavior of the applications in the Application Layer  14  and further customize the portals in the User Access Portal Layer  12 . This configuration and customization includes, but is not limited to, adapting the look and feel of the application through each portal and the nature of the data filters applied to the data at the Application Layer  14 . However, it is contemplated that the system may be customized based upon any data parameter relative to user experience and human computer interaction. 
         [0032]    The eMail Portal (eMP)  26  is primarily used to provide query access to MMP data through eMail messages. It is contemplated that the MMP  10  is compatible with proprietary email servers such as Microsoft Exchange and the like and also compatible with any conventional email servers such as Yahoo Mail or Microsoft Hotmail and the like. The user&#39;s identity is recognized and authenticated through the originating email address and the query result is organized in the form of a return email to the user. However, it is contemplated that the eMP  26  may organize the query result in email form and subsequently transmit the email to any email address determined by the user or content provider. 
         [0033]    The Downloadable Application Portal (DAP)  28  is used to communicate with the MMP through an application that is downloaded to and resides on the mobile device, and is capable of accessing any communications network such as the Internet  36 . The mobile device may access the Internet  34  to communicate with the DAP  28  through various modes of connectivity such as, but not limited to, WAP, GPRS, EDGE, 3G, 4G, WiFi, or WiMAX and the like. The communication between the downloaded application and the DAP  28  may occur over standard protocols such as HTTP (Hyper Text Transfer Protocol) or a proprietary protocol that both DAP  28  and the downloaded application are compatible with. The downloaded application provides functions similar to WAPP  22 , but since the downloaded application runs on the mobile device it may use the local operating system of the mobile device to render local user interface elements for the user, and thereby make interactions easier, faster, and more efficient. It is contemplated that third party providers may develop downloadable applications. 
         [0034]    In another embodiment of the present invention the present system includes an Advertiser Portal (AP)  30 . The AP  30  is a specialized web based online portal that is dedicated to a group of users known as Advertisers. The Advertiser group may access the AP  30  through utilities built into the AP  30  and deposit various messages having a variety of content such as commercial, informational, educational and the like. These messages are then used by the Advertisement Selection Engine (ASE) module  36  to provide timely and related messages to user based on the nature of their interaction while they are using one of the VP  18 , SMSP  20 , WAPP  22 , WP  24 , eMP  26 , or DAP  28  portals. The ASE  36  is implemented and stored on the Application Layer  14 . When needed, the messages are converted to an appropriate form for each portal, Voice for VP  18 , Text Message for SMSP  20 , WAP or Mobile Web message for WAPP  22 , email message for the eMP  26 , online message for WP  24 , and application message for DAP  28 . Therefore, the AP  30  and ASE  36  collectively work to direct appropriate advertisements to users. 
         [0035]    The Application Layer  14  is the core of the MMP  10 . In this regard, the Application Layer  14  is where the business logic is programmed and implemented to generate the query results for the system. The Application Layer  14  includes a plurality of all purpose services, engines, or programs that are utilized by other layers of the MMP  10 . In the present embodiment, the Application Layer  14  comprises a Text to Speech Engine  38 , an Automatic Speech Recognition Engine  40 , a Data Search Engine  42 , a Data Comparison Engine  44 , an Advertisement Selection Engine  36 , a Network Gateway Engine  46 , a Collection Management Engine  48 , a User Customization and Configuration Engine  50 , a Purchase Management Engine  52 , and Friend&#39;s List Management Engine  54 . However, it is contemplated that the Application Layer  14  may store and run any programmable computer instruction such as a program, service, or engine to manipulate, store, or interact with data. 
         [0036]    The Text to Speech Engine (TTS)  38  converts the written text to human speech. The TTS  38  is used primarily by to interact with the user through the VP  18 . For example, a user initiated written query may trigger a result from the data source that is converted to spoken words through TTS  38 . The result is subsequently communicated back to the user through VP  18 . The TTS  38  may also be used by the ASE engine  36  to convert written messages to spoken messages for use with the VP  18 . 
         [0037]    Automatic Speech Recognition Engine (ASR)  40  performs a reverse function of TTS  38 . In this regard, the ASR  40  is used to convert spoken query words to text. The ASR  40  is primarily used in recognizing the spoken words coming through the VP  18  and converting it to text so other applications in the Application Layer  14  can process the query. 
         [0038]    The Data Search Engine  42  performs the search on the online data based on the user query or other application initiated queries. Data Comparison Engine  44  performs various forms of comparison amongst data components based on user or other application generated queries. 
         [0039]    Advertisement Selection Engine (ASE)  36  selects the most appropriate advertisement message from a pool of messages, based on a variety of criteria. In the present embodiment, the ASE  36  selects the appropriate message based on relevance of the message to the user query by matching the message-supplied key words against the user query criteria, Date and Time of the query, Type of input portal (VP  18 , SMSP  20 , WAPP  22 , WP  24 , eMP  26 , DAP  28 ), User supplied customization and configuration parameters, or User demographics. However, it is contemplated that the ASE  36  may use any user criteria based on any data parameter to select messages for a particular user. 
         [0040]    The Network Gateway Engine (NGE)  46  provides the necessary authentication, translations, filtering, and caching to efficiently communicate with an external data source  56 . The external data source  56  may be any data source such as an internet based data source or the like. In the present embodiment, the NGE  46  supports XML data format to efficiently communicate with external data sources. However, it is contemplated that the NGE may support any data format to effectively communicate with external data sources. 
         [0041]    The Collection Management Engine (CME)  48  manages user interactions or queries. Specifically, all user queries are kept in a user accessible ‘user history log’. These interactions may be placed in variety of collections. It is contemplated that collections are organized with conventional data structures such as matrices, arrays, lists, and the like. Collections are applied to any number of “items of interest” to the user, such as collections of products, collections of events, collections of blog posts or RSS feeds, or the like. It is contemplated that the CME  48  may comprise any number of collections. In the present embodiment, the CME  48  includes the History Collection and the Bookmarks Collection. The History Collection stores the history of all the user&#39;s transactions. The Bookmarks Collection collects the history of the transactions the user would like to highlight as ‘bookmarked’. 
         [0042]    The user manages items in these collections or moves them into other customized or predetermined collections through the Web Portal  24 . For example, in the case of products, the user moves or copies products from the History Collection or the Bookmarks Collection into a custom list such as a “Wishlist”, “Favorite Gadgets List”, “Books Read List”, “Movies I Own List”, or the like. It is contemplated that the user may employ any list or customized collection to organize data. Collections and lists advantageously permit the user to organize and manage data. Collections are user managed through accessing the CME  48  from any of the VP  18 , SMSP  20 , WAPP  22 , eMP  26 , WP  24 , or DAP  28  portals. However, it is preferred that collections are managed through WAPP  22 , WP  24 , or DAP  28  for efficiency. Additionally, the CME  48  allows the user to attach specific functions to any transaction log. For example, a user may wish to attach a personalized Voice Note to a transaction log, or attach an alert to a transaction log when the condition of that log item changes in the source data. An example would be attaching an alert function to a searched product and requesting an alert message when the price of the product falls below a threshold. However, it is contemplated that an alert may be associated with any data parameter associated with the present system. 
         [0043]    The User Customization and Configuration Engine (CONF)  50  customizes the look and feel of the data presented to the user through any of the portal as well as changing the applications behavior by managing them through the CONF engine  50 . Thus, the CONF  50  allows the user to personalize the present system to provide greater comfort and increase efficiency. The Purchase Management Engine (PME)  52  coordinates the purchase function between the MMP  10  and sales transactions with merchants. The Friends List Management Engine (FLME)  54  supports group messaging using the eMail, Voice, Text Message, or Downloadable Application. It is contemplated that a user on this platform can define one or more Friends (or Buddy) lists. At times the user can decide to send to such list or lists a message (written or spoken) or to send one or more items from the users history. The FMLE module  54  is responsible for that interaction as well managing the user&#39;s friends list. The Friends Lists can be stored either locally, or can be retrieved remotely from a third-party social networking site such as Facebook or the like that makes the friends map of a user available through certain APIs (Application Programming Interfaces). 
         [0044]    The MMP  10  abstracts the above mentioned engines or services into a set of XML objects with which other applications can interact. However it is contemplated that any programmable data structure may be used to organize the data. Other “application connectors” in the Application Layer  14  can be specifically designed to perform a business function, interface with an external third party application, make queries, and pass the data to the base services to be presented to the user via various portals. As an example, consider an online calendar service such as Google Calendar or the like which publish data in a certain format. A user can create an application connector for Google Calendar that retrieves the published data and pass it to the Voice or Text Message engines so that the user can access the same calendar data by placing a phone call or sending an SMS. Conventionally, a user would typically use the Internet  34  from a PC to look at his or her calendar, but the present system provides a robust means to access data in a convenient technological methods. 
         [0045]    It is contemplated that the various online data tools such as web blog entries using an RSS feed, web journaling, and the like may be employed by the system. Such application connectors use various utility engines described above to implement their logic. Users can use the Web Portal  24  to configure, personalize, and enable/disable the connectors for their preferred applications. Developers can use the published programmable scripts using XML objects, APIs, and the like to create more application connectors. 
         [0046]    The Database and Data Access Layer  16  is where the internal data, customer information, customization information, Advertising messages and business data is deposited. However it is contemplated that any data or metadata referenced by the user or third party content providers may be stored on the data layer. It is contemplated that certain data components of the business data may be stored externally and may be accessed through the NGE. The internal data is managed through Transaction Database Manager (TDM)  58 , which is responsible for recording user interactions, preferences and logs, and the Audio Database Manager (ADM)  60  which responsible for managing the audio files created by users or advertisers. There is a single Read/Write copy of these databases with multiple Read-only copies for redundancy and efficiency of access. However the Database and Data Access Layer  16  may employ any database structure or data organizational tool having various Read/Write attributes. 
         [0047]    The particulars shown herein are by way of example and for purposes of illustrative discussion of the embodiments of the present invention only and are presented in the cause of providing what is believed to be the most useful and readily understood description of the principles and conceptual aspects of the present invention. In this regard, no attempt is made to show structural details of the present invention in more detail than is necessary for the fundamental understanding of the present invention, the description taken with the drawings making apparent to those skilled in the art how the several forms of the present invention may be embodied in practice.