Patent ID: 12244752

DETAILED DESCRIPTION OF THE DISCLOSURE

The various aspects of the subject disclosure are now described with reference to the drawings, wherein like reference numerals correspond to similar elements throughout the several views. It should be understood, however, that the drawings and detailed description hereafter relating thereto are not intended to limit the claimed subject matter to the particular form disclosed. Rather, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the claimed subject matter.

As used herein, the terms “component,” “system” and the like are intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a computer and the computer can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers or processors.

The word “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs.

Furthermore, the disclosed subject matter may be implemented as a system, method, apparatus, or article of manufacture using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer or processor based device to implement aspects detailed herein. The term “article of manufacture” (or alternatively, “computer program product”) as used herein is intended to encompass a computer program accessible from any computer-readable device, carrier, or media. For example, computer readable media can include but are not limited to magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips . . . ), optical disks (e.g., compact disk (CD), digital versatile disk (DVD) . . . ), smart cards, and flash memory devices (e.g., card, stick). Additionally it should be appreciated that a carrier wave can be employed to carry computer-readable electronic data such as those used in transmitting and receiving electronic mail or in accessing a network such as the Internet or a local area network (LAN). Of course, those skilled in the art will recognize many modifications may be made to this configuration without departing from the scope or spirit of the claimed subject matter.

Referring now to the drawings wherein like reference numerals correspond to similar elements throughout the several views and, more specifically, referring toFIG.1, the present disclosure will be described in the context of an exemplary communication system10that can be used to facilitate communication between a hearing user22and an assisted user24(e.g., a hearing impaired user) where the communication is similar to the type of communication that occurs between two hearing persons via a telephone connection. The hearing user22may use a normal telephone to link into the system via a PSTN or other type of telephone line or communication link to make a call or receive a call from the assisted user24.

Exemplary system10includes a captioned telephone device14, a wireless router device16and a tablet type computing device20located at the residence or some other location (e.g., an office, place or work, etc.) associated with assisted user24and within wireless broadcast range of router16. In addition, system10includes a relay or call center12that is typically remote from the location at which router16is located.

In at least some embodiments, captioned telephone device14is a two line type caption device that is linkable by phone (e.g., PSTN) or other communication link (e.g., internet) on a first line to the hearing user's device18. In at least some embodiments device14may be used like a normal telephone device to facilitate a conventional telephone conversation between two hearing users, one using device18and the other using device14. To this end, referring also toFIG.2, device14includes, among other components, a keyboard32for entering a phone number or other information and a handset34that can be used to receive a hearing user's voice broadcast from a speaker and to receive a local user's voice signals via a microphone built into the handset (not separately labeled). In at least some embodiments, while not illustrated, device14includes a speaker and a microphone that can be used to broadcast a hearing user's voice signal and to receive an assisted user's voice signal to be transmitted on to the hearing user at device18. In some cases the speaker can be set to have a relatively high volume to help an assisted user hear the hearing user's voice signal. Voice signal attributes other than volume can be adjusted to increase a specific user's ability to hear (e.g., attributes like pitch and tone can be adjusted to meet specific and instantaneous user preferences).

Referring still toFIG.2, in addition to the components described above, captioned telephone device14also includes a large (e.g., 7 to 9 inch diameter) display screen30which can be used to interface with device14by providing screen shots of content and virtual tools (see35,44,42, etc.) as well as to provide text or captioning corresponding to words spoken by a remote hearing user22to an assisted user using device14. To this end, see the exemplary caption text at38that corresponds to words spoken by hearing user22in the present example. The text is presented substantially in real time (e.g., with less than 1-4 seconds delay in most cases and often times faster) as a remote hearing user speaks. The exemplary display screen shot35inFIG.2includes additional useful information including a header at37which indicates the identity of the remote user22when available as well as a current time and date and a call timer which indicates the period associated with an ongoing call in minutes and seconds.

In at least some cases the screen30is touch sensitive so that virtual control icons or buttons can be presented on the screen and the screen can be used to sense when a user selects one of the virtual buttons. Exemplary virtual buttons include a scrolling tool at40enabling a user to scroll through content as that content is presented on the screen30, a pause button44selectable to pause the captioning text service and a text resizing tool at42useable to increase or decrease the size of text presented in the captioned phrases at38. In at least some cases the keyboard may include a special key36that is selectable to start and stop the captioning service at any time during an ongoing call or even prior to commencement of a call. For instance, while a conventional voice call is ongoing, an assisted user may recognize that she needs captioning as she cannot make out at least some words uttered by a hearing user. In this case, the user may select button36to start a captioning service. At some point while captioning is being provided during an ongoing call, a user may recognize that she does not need the captioned text and may select button36to turn off the captioning service.

Referring again toFIG.1, device14has an Ethernet port or a wireless transceiver that can be used to link to a local area network router device16or some other intermediate device that in turn is linked to the router device. The router16is linked via an internet or other communication link to the relay center12. When captioning is not required, device14is, in at least some cases, not linked to router16and instead conventional phone communications occur between a hearing user's device18and the assisted user's device14. When captioning is required, device14links to router16and causes a connection to relay12over which the hearing user's voice signal is transmitted. Here, when device14is used without a remote tablet device20, device14cancels an assisted user's voice signal from the voice signals transmitted to the relay12so that only the hearing user's voice signals arrive at the relay12to be converted to text. By cancelling the assisted user's voice from the signal sent to the relay, the relay can transcribe the entire received voice signal without attempting to separate the assisted user's voice from the hearing user's voice signal and the transcription process is rendered more simple.

Referring still toFIG.1, relay center12comprises a system that can receive voice signals via the internet or other communication connection and transcribe the voice signals into text. In at least some cases, center12may employ call assistants or relay operators for transcribing voice signals to text. To this end, in at least some cases, each relay call assistant will be equipped with a computer having a display screen, a keyboard, a head set including an earphone and a microphone and perhaps other input devices. The computer may operate voice-to-text transcription software that is trained to the voice of the call assistant to transcribe voice signals from the call assistant to text. The hearing user's voice signal may be presented to the ear of the call assistant and the call assistant may revoice the hearing user's voice signal into the headset microphone. The computer then uses the transcription software to transcribe the call assistant's voice to text which is, in at least some cases, presented on the display screen linked to the call assistant's computer. The call assistant can then correct any transcription errors in the text presented on the display. The transcribed text is transmitted back to router16via the internet or other communication link which transmits the text on to device14to be displayed on screen30. In at least some cases the transcribed text is transmitted to device14only after error correction has occurred. In other cases, the transcribed text is transmitted immediately upon generation by the software including errors and then any corrections to the transcribed text are subsequently transmitted and used to make in line or other corrections to the prior transmitted text so that transcription is presented to the assisted user as quickly as possible.

In other cases, instead of call assistants transcribing the hearing user's voice signal to text, automated transcription software running on a server or computer at relay12may receive a hearing user's voice signal and automatically transcribe that signal to text. Here, the automatically transcribed text may be sent immediately back to device14via the internet link to be presented via device14. In other cases, a call assistant may correct the software transcribed text and corrections may either be made prior to transmitting the software transcribed text to device14or after that text is transmitted in which case the corrected text would be transmitted to device14and used to make in line or other types of corrections at device14. In still other cases other combinations of automated transcription software and call assistant activities may be used at relay12to transcribe hearing user's voice to text. U.S. patent application Ser. No. 14/632,257 which is titled “Semi-automated Relay Method And Apparatus” and which was filed on Feb. 26, 2015 describes several different relay systems where software or a call assistant and software combination are used to transcribe a hearing user's voice to text, any of which may be used in system10, which is incorporated herein in its entirety by reference.

Referring yet again toFIG.1, consistent with that least some aspects of the present disclosure, an assisted user24may own or have access to a portable tablet or other type of portable computing device20. Here, device20may be any type of portable computing device that includes a display screen, a wireless transceiver and a processor that is capable of performing the methods and processes described herein. For instance, device20may include a laptop computer, a tablet type computing device like an I-Pad, an Android based tablet device, etc., a smart phone or other type of computing device. Hereinafter, unless indicated otherwise, while device20may take many different forms, the phrase “tablet device” will be used to refer to device20or any other type of portable computing device used in a system10.

Referring toFIGS.3and4, exemplary tablet device20is illustrated which includes, among other components, a touch sensitive display screen50, a processor63, a transceiver65, one or more speakers69, a microphone59and, in at least some cases, one or more hardware interface buttons67. Processor63is linked to each of the screen50, transceiver65, buttons67, microphone59and speaker69. Wireless signals from router16are received via transceiver65and wireless signals are also transmitted from processor63to router16via transceiver65. In addition, processor63can communicate with other wireless devices such as an assisted user's headset82(see alsoFIG.1) that includes earphones and a microphone to transmit a hearing user's voice signal to and receive an assisted user's voice signal from device82, a wireless speaker or microphone, etc.

As seen inFIG.1, a wireless (e.g., WIFI) link may be established between router16and client device20so that device20can be networked to the captioned telephone device14as well as to relay12. Here, client device20can be used as a wireless remote interface to captioned device14and, in at least some cases, may support substantially all or all of the assisted user interface input and output functionality that is provided by host device14. Thus, for instance, when a call is received at captioned device14, in addition to announcing the call at device14via a ring tone and some type of visual notice on screen30or a flashing light (not shown) associated with device14, device14may transmit a signal to router16and on to client device20causing device20to announce the incoming call in a similar fashion. To this end, see, for instance,FIG.4where a screen shot51includes an announcement window at52indicating that an incoming call has been received by device14. The announcement includes, where possible, the name of the remote hearing person calling device14and a phone number associated with the device18used to initiate the call (see54inFIG.4). In addition, the announcement includes selectable virtual on screen icons or buttons56and58for answering the call and ignoring the call, respectively. Here, to answer an incoming call via device20, a user simply selects icon56and to stop the announcement and end the call, the user simply selects icon58.

In addition to announcing via window52, device20may generate a ringing or other signal via speaker69. In addition, device20may control the background of screen50that surrounds window52to clearly indicate an incoming call by, for instance, generating a red background color as indicated by the left up to right cross hatching shown inFIG.4.

In other cases the incoming call notice may be more discrete such as a simple banner (see53inFIG.4) at the top of the screen50to announce an incoming call. In still other cases device20may generate different types of incoming call announcements depending on how device20is currently being used by an assisted user. For instance, if the assisted user is currently using an application on device20for some other purpose (e.g., reading a news article, playing a game, etc.), a relatively discrete announcement like banner announcement53may be presented as the user, staring at the screen, should be able to notice the announcement when it arrives. Here, if the user is not currently using an application on device20, the announcement may be less discrete such as the window form shown at52inFIG.4to more strongly call the user's attention to the incoming call. In at least some cases sensors already included in device20may be able to discern the type of notice to present as a function of immediate use by a device user. For instance, an accelerometer or other sensor in device20may sense when the device is being moved and may discern instantaneous use by a user therefrom. As another example, a camera in device20may obtain an image of the user when an incoming call is received and the device20processor63may analyze the image and determine that the user is looking directly at the screen50so that a more discrete banner or other type incoming call announcement can be generated.

When the device20user receives a call and selects icon58inFIG.4to ignore the call, device20transmits a signal via router16to host device14indicating that the user does not want to answer the call. In response to the received ignore signal, host device14may link the incoming call to a voice mail or call answering service. Here, the call answering service may answer the call, present a voice message to the calling party22and request that the calling party leave a voice mail message. When a voice mail message is provided, the voice signal may in some cases be stored solely as a voice mail message. In other cases, the voice mail message may be automatically presented to relay12and converted to text. The text may be transmitted back to device14to be stored locally along with the voice mail for subsequent access by the assisted user82. In other cases the voice and transcribed text may be stored via a server at relay12to be accessed subsequently by user24. In some cases where voice mail transcription is supported, transcription may occur immediately when a voice is recorded while in other cases the transcription may occur at some subsequent time as immediate transcription of a stored voice mail is not typically necessary (e.g., transcription 10 minutes after a hearing user's voice signal is recorded is usually acceptable).

Because transcription of a voice mail is typically not time sensitive, in at least some cases, voice to text transcription software may be used to generate an initial text version of a voice signal. Here, in some cases the initial text version may be presented to an assisted user when the user accesses the voice mail associated therewith. In some cases, if the initial version of the text is insufficient (e.g., clearly not accurate or includes excessive errors), the assisted user may request a more accurate version at which point the original voice signal may be presented to a call assistant at relay12for full call assistant transcription or at least for correction of the initial text version. Corrected or call assistant transcribed text may then be transmitted back to the assisted user device14or device20to be presented to the assisted user.

When device20user receives a call and selects answer icon56inFIG.4, a signal is transmitted back through router16to captioned host device14. The signal received at device14causes device14to initiate a link via router16to relay12to commence captioning in at least some embodiments. In addition, the signal received at device14causes device14to answer the incoming call (e.g., go off hook) and start transmitting voice signals between the hearing user's device18and the assisted user's tablet20via router16. Here, the hearing user's voice signal received at device14is routed through router16to each of relay12and tablet20. When the hearing user's voice signal is received at relay12, the signal is converted to text and the text is transmitted back to device14via router16. In at least some cases text received at device14will always be presented via device14. In other cases where an assisted user is using device20to view text, the text will not be presented via device14.

When device14receives text from relay12and an assisted user is using tablet20for captioning, device14transmits the received text via router16to device20. Thus, device20receives both the hearing user's voice signal as well as the transcribed text associated with the hearing user's voice signal. Because it takes at least some time to transcribe the voice signal to text, the hearing user's voice signal is typically received at tablet20at least some time prior to receiving the text and therefore there is some delay between voice broadcast and text presentation by tablet20. For this reason, in at least some cases device14or tablet20or even relay12in some cases may delay the voice signal for a short duration (e.g., 2 or less seconds) so that the text and associated voice signal are more synchronized. In still other cases it is contemplated that time stamps may be associated with incoming voice signals (e.g., a separate time stamp for each word discerned in a received voice signal) and text may be associated with each time stamp so that substantially precisely synchronized text and voice can be presented. Here, delays in voice and text presentation may be presented at the ends of sentences as opposed to between words so that the sentences make more sense and can be presented at the pace of the original utterances by the hearing user.

Voice signals from assisted user24received via microphone59are transmitted from tablet20via router16to device14and on to the hearing user's device18to be broadcast to the hearing user.

Thus, referring toFIG.5, when client device20is used as a remote captioning device, an additional IP connection80via a home or business network (e.g., a router16) is established to initiate a two way audio stream (e.g., PCM via UDP), one way captioning to device20via TCP (or some other suitable protocol) and two way command/control signal transmission between host device14and client device20.

Referring yet again toFIG.1, in at least some embodiments the assisted user24may use a wireless or tethered headset82including earphones and a microphone. In this case, the assisted user's voice signal would be obtained via the headset microphone and transmitted to tablet20to be forwarded on to device14and then on to the hearing user's device18. The hearing user's voice signal would be transmitted from tablet20to the headset earphones to be broadcast to the assisted user. Text from relay12would still be presented via the tablet20display.

Referring now toFIG.6, an exemplary screen shot that may be presented via tablet screen50during ongoing captioning is shown. The exemplary shot includes information and tools that are similar to the information presented via the captioned device screen shown inFIG.2. To this end, information descriptive of an on going call is presented at37aincluding the name of the hearing user participating in the call, the hearing user's telephone number and, where available, an image of the hearing user. Transcribed text is presented at38aand a text scrolling tool is presented at30a. Pause and text size control icons are presented at44aand42a, respectively. Comparing the screen content inFIG.6with the content inFIG.2, similar content is indicated by the same numerical label, albeit including an “a” qualifier inFIG.6. By presenting content in a similar format and with a similar look and feel, user's familiar with the content presentation on the captioned device screen30will be automatically familiar with the similarly presented content on tablet screen50.

Referring still toFIG.6, in addition to the other information presented as described above, screen shot77also includes an “End Call” icon40for ending an ongoing call. When icon40is selected, tablet20may exit a captioning application immediately. In addition, upon ending a call, tablet20transmits a control signal to caption device14causing device14to terminate the telephone link to the hearing user's device18and terminate the link to relay12.

In addition to using tablet20to field an incoming call, tablet20may be used to place a call that provides captioning service. To this end, seeFIG.7where an exemplary captioning application main screen shot61is shown that includes several different virtual control icons62,64,66,68and70that can be used to access different captioning application features. Icon62is a “Call” icon which, when selected, allows an assisted user to initiate a captioned call to another person (e.g., the hearing user22inFIG.1). For instance, when icon62is selected, in at least some embodiments the exemplary screen shot71shown inFIG.8may be presented. Exemplary screen shot71includes a field72and a virtual numeric keyboard74for entering a telephone number. In addition, in at least some cases contact information for other persons that the assisted user has previously stored or has recently communicated with may be presented as at78. InFIG.8the exemplary contact information includes a name and image of teach of several different contacts. Any of the images may be selected to place an associated phone number in field72as an initial step to commencing an outgoing call. Other formats for presenting contact information78are contemplated. Once a number is entered into field72, phone icon76may be selected to initiate a call.

Referring again toFIG.1, to initiate a call, device20transmits a call request via router16to caption device14. Upon receiving the call request, device14dials out on the telephone line to the target hearing user's device (e.g. the phone associated with the number entered or selected by the assisted user). In at least some cases if a hearing user answers an outgoing call, caption device14automatically controls router16to link to relay12to provide the hearing user's voice signal to relay12and to receive transcribed text back from relay12to be presented to the assisted user. The returned transcribed text received at device14is routed to tablet20via router16to be presented to the assisted user24. Again, the call can be terminated by the assisted user at any time by selecting the end call icon40shown inFIG.6.

Referring again toFIG.7, answering machine icon64may be selected to access stored voice mails and associated transcribed text. In at least some embodiments voice mails and associated text may be stored in a database linked to host device14or accessible via router16. History icon68may be selected to access a list of recent calls or call attempts to render those contacts easily accessible. Favorites icon70can be selected to access a list of an assisted user's favorite contacts for convenience. Contacts icon66may be selected to access stored contact information. In some cases the contact information is stored by a relay or other system server or device that is separate from portable tablet20so that selection of icon66causes

In at least some cases it is contemplated that an assisted user will not need or want captioning when a call is initially answered but may want to turn on a captioning service at some time during an ongoing call. Nevertheless, in this case, the assisted user would often like the ability to answer an incoming call or initiate an outgoing call at a location remote from the host device14. For this reason, in at least some cases, calls may be answered or initiated using client device20linked to host device14without captioning and the captioning may only be turned on at some subsequent time. To this end, if the answer icon56is selected inFIG.4, at least initially, client device20may send a signal via router16to host device14to answer the call and start an audio only communication session while presenting a “Caption” icon option99as shown inFIG.9. Here, as the audio only call is progressing, an assisted user may select icon99to cause host device14to link via router16to relay12to commence the captioning service. Again, text would be transmitted via router16to host device14and from host device14through the router16to the client device to be presented on the client device screen50.

In other cases, when a tablet20is paired with or associated with a caption device14, device14may tag a hearing user's voice signal with a virtual ID associated with tablet20so that when text is transmitted back from relay12to be presented to the assisted user, the text can be sent through the router16directly to tablet20via virtual ID associated therewith without having to first be received at the host device14. In this case, the text may or may not be transmitted to device14and may or may not be presented via device14even if transmitted thereto. Thus, for instance, when a call is received at device14, an incoming call announcement (see52inFIG.4) may be presented via device20and when answered, device14may link to relay12to start the captioning process. Here, the hearing user's voice is transmitted to relay12along with the assisted user's tablet virtual ID or address and relay transmits the transcribed text via the internet or other communication link through router16directly to tablet20to be presented via screen50.

In still other cases, although not shown, the host device14may include a built in router so that the Ethernet/wireless link shown inFIG.1is not required. Here, host device14would simply link via the internet or other communication link to relay12and wirelessly to device20via a local area or wide area network.

In yet other cases device20may include a personal hotspot or other type of a router capability and router16may again not be needed. To this end, seeFIG.10that is similar toFIG.1, albeit where the router16is replaced by a router internal to device20. In this case, when relay services are required and an assisted user is using a tablet or other portable device20with a built in routing capability, device14may provide the hearing user's voice signal to device20which in turn transmits that voice signal to relay12while also broadcasting that voice to the assisted user24. Relay12again provides transcribed text which is transmitted to tablet20to be displayed on the tablet screen50. The assisted user's voice signal in this case is also transmitted from tablet20to caption device14and then on to the hearing user's device18. Here, if an assisted user wants to switch over to viewing captions on host device20, the user may initiate that process which would in turn cause captions to also be presented via the host device display. Here, the phone link between the assisted user's host device and the hearing user's device is a direct phone, internet or other type of communication link while device20handles back end communication with relay center12.

In still other cases, instead of providing a full featured host captioned device, a relatively less expensive captioned device may be provided so that communication components that are provided via the client device20are not duplicated via the host device. For instance, seeFIG.11that is similar toFIG.1except that the full featured captioned device14has been replaced by a captioned device brick component100(hereinafter “brick device100”). Brick device100has all of the components and capabilities of host device14except for the display screen and handset (including eh speakers and microphone). Here, brick device100has an assigned phone number that can be used to receive an incoming call from a remote user's device18. As in the case of host device14described above, device100can be controlled to link via a home network including router16to client device20to facilitate an audio only conversation between users of devices18and20or an audio/captioned communication session between device18and20users. For instance, when an initial call comes in to brick device100, device100may link to client device20via router16and cause device20to announce the incoming call (see againFIG.4). When the call is answered (e.g., icon56is selected inFIG.4), brick device100may facilitate an audio only communication between the device18and24users at least initially, client device20presenting a Caption button option as inFIG.9. If button99is selected, client device20would send a command to host brick device100to transmit the hearing user's voice signal from device18to relay12via router16and the second link (e.g., the Internet link InFIG.11). Relay12transmits captioned text back to the brick device100which in turn transmits that text to client device20via router16to be presented on the client device display screen50. In this case, because brick device50does not include a display, speakers or a microphone, the assisted user does not have the option to use device100as a direct input and output interface and instead would only use the client device20for those purposes.

In at least some cases a host device14and client device20will be used in a home or other facility where more than one phone device shares the phone line used by the host device14to link to remote user devices (e.g.,18). In this case, in at least some cases host device14will monitor when any phone linked to the incoming line if off hook and may transmit a signal indicating the off hook state to client device to be announced (e.g., presented as a visual queue) to a user of device20so that the device user understands the current state of the in coming line. Here, for instance, if the wife of an assisted user is using a different phone in the assisted user's home to participate in a call, the assisted user may not want to disturb that call and therefore the notice of phone line state would be desired.

In at least some cases it is contemplated that an assisted user may join an ongoing call that involves some other phone device in the assisted user's home or other associated space using a client device20and the assisted user may or may not add captioning to the joined call. For instance, in some cases when an ongoing call is progressing, client device20may present the option to “Join Call” via the device display50which, when selected, causes client device20to transmit a command to host device14to go off hook and connect the audio portion of the ongoing call to device20via router16. If captions are not started immediately, when the assisted user wants to start the caption service, the user may select a caption icon as inFIG.9as described above.

Support case where text presented on tablet20and then moved to device14.

Support case where text presented via phone device remotely and then when proximate device,14, presenting text automatically via device14to take advantage or larger screen, speaker, etc.

While the invention may be susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and have been described in detail herein. However, it should be understood that the invention is not intended to be limited to the particular forms disclosed. For example,

Thus, the invention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention as defined by the following appended claims.

To apprise the public of the scope of this invention, the following claims are made: