Glass type mobile terminal

A glass type mobile terminal including a transparent screen; a frame configured to secure the transparent screen in front of a user's eyes wearing the glass type mobile terminal; a camera mounted to the frame and configured to photograph an image in front of the user's eyes; a memory; an image recognition unit configured to extract information from the image photographed by the camera; and a controller configured to compare the extracted information with related information stored in the memory, and display the related information to the transparent screen on the transparent screen along with the captured image.

CROSS-REFERENCE TO A RELATED APPLICATION

Pursuant to 35 U.S.C. §119(a), this application claims the benefit of earlier filing date and right of priority to Korean Application No. 10-2012-0140408, filed on Dec. 5, 2012, the contents of which are hereby incorporated by reference herein in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a glass type mobile terminal including a camera and a transparent screen, which can acquire and use information easily with leading a daily life.

2. Discussion of the Related Art

Generally, terminals can be classified into mobile terminals and stationary terminals. In addition, the mobile terminals can be further classified into handheld terminals and vehicle mount terminals. Further, a mobile terminal can perform various functions such as data and voice communications, capturing images and video via a camera, recording audio, playing music files and outputting music via a speaker system, and displaying images and video on a display.

Recently, interest in wearable mobile terminals is rising for user convenience and research is in progress on various types of mobile terminals mounted to glasses, bracelets and clothes.

SUMMARY OF THE DISCLOSURE

An object of the present invention is to provide a glass type mobile terminal including a camera and a transparent screen, which can acquire and use information easily helping the user in their daily life.

To achieve these objects and other advantages and in accordance with the purpose of the invention, as embodied and broadly described herein, a glass type mobile terminal includes a transparent screen; a frame configured to secure the transparent screen in front of the user's eyes; an audio output module configured to transmit sound to the user; a camera mounted to the frame to photograph an image in front of the user's eyes; a memory configured to store input/output information therein; an image recognition unit configured to extract information from the image photographed by the camera; and a controller configured to perform at least one of storing the information extracted from the image in the memory, comparing the extracted information with related information stored in the memory, and outputting the related information to the transparent screen or the audio output module.

The image recognition unit may extract the information when the camera photographs the same image for more than a preset time period. Alternatively, the image recognition unit may extract information related to the object, When one object occupies a preset range of areas possessed by the image.

According to a first embodiment of the present invention, the image may include a calendar, and the extracted information may include the year, month and date of the calendar and a memo written in the calendar.

According to a second embodiment of the present invention, the image may be a memo and the information may be a character written on the memo. The controller may display stored-information corresponding to the character written in the memo in a preset level or more on the transparent screen.

According to a third embodiment of the present invention, the image may include an object and the information may be an image of the object photographed at one or more angles, and the controller may compare the photographed image with an image of a designated object stored in the memory and determines whether the photographed object corresponds to the image of the designated object.

At this time, the glass type mobile terminal may further include a location information module configured to recognize the user's location, wherein the controller records the user's location in the memory as a log record for the designated object, When the photographed object corresponds to the designated object and the user can read the log record.

Alternatively, the controller may output the stored information related to the designated object to the transparent screen or output the stored information via the audio output module, When the photographed image corresponds to the designated object.

The glass type mobile terminal may further include a microphone configured to convert a sound near the glass type mobile terminal into a digital signal, wherein the image is a bill or coin, and the information is the amount of bills or coins and the controller compares voice information related to the amount of the money acquired by the microphone with the amount of the bills or coins to calculate the change and outputs the calculated change to the transparent screen or outputs the calculated change via the audio output module.

DESCRIPTION OF SPECIFIC EMBODIMENTS

In the following detailed description, reference is made to the accompanying drawing figures which form a part hereof, and which show by way of illustration specific embodiments of the invention.

As used herein, the suffixes ‘module’, ‘unit’ and ‘part’ are used for elements in order to facilitate the disclosure only. Therefore, significant meanings or roles are not given to the suffixes themselves and it is understood that the ‘module’, ‘unit’ and ‘part’ can be used together or interchangeably.

FIG. 1is a block diagram of a glass type mobile terminal100in accordance with an embodiment of the present invention.FIG. 1shows the glass type mobile terminal100according to one embodiment of the present invention includes a wireless communication unit110, an input unit120, a sensing unit140, an output unit150, a memory160, a controller180, a power supply unit190and the like.FIG. 1shows the glass type mobile terminal100having various components, but it is understood that implementing all of the illustrated components is not a requirement. Greater or fewer components may alternatively be implemented.

In the following description, the above elements of the glass type mobile terminal100are explained in sequence.

First of all, the wireless communication unit110typically includes one or more components which permits wireless communication between the glass type mobile terminal100and a wireless communication system or network within which the glass type mobile terminal100is located. For instance, the wireless communication unit110can include a broadcast receiving module, a mobile communication module112, a wireless internet module113, a short-range communication module114, a position-location module115and the like.

The mobile communication module112transmits/receives wireless signals to/from one or more network entities (e.g., base station, external terminal, server, etc.) via a mobile network such as GSM (Global System for Mobile communications), CDMA (Code Division Multiple Access), and WCDMA (Wideband CDMA) and so on. Such wireless signals may represent audio, video, and data according to text/multimedia message transmission and reception, among others.

The wireless internet module113supports Internet access for the glass type mobile terminal100. This module may be internally or externally coupled to the glass type mobile terminal100. In this case, the wireless Internet technology can include WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), GSM, CDMA, WCDMA, LTE (Long Term Evolution) etc.

Wireless internet access by Wibro, HSPDA, GSM, CDMA, WCDMA, LTE or the like is achieved via a mobile communication network. In this aspect, the wireless internet module113configured to perform the wireless internet access via the mobile communication network can be understood as a sort of the mobile communication module112.

The position-location module115identifies or otherwise obtains the location of the glass type mobile terminal100. If desired, this module may be implemented with a global positioning system (GPS) module. According to the current technology, the GPS module115can precisely calculate current 3-dimensional position information based on at least one of longitude, latitude and altitude and direction (or orientation) by calculating distance information and precise time information from at least three satellites and then applying triangulation to the calculated information. Currently, location and time informations are calculated using three satellites, and errors of the calculated location position and time informations are then amended using another satellite. Besides, the GPS module115can calculate speed information by continuously calculating a real-time current location.

Referring toFIG. 1, the input unit120is configured to provide audio or video signal input to the glass type mobile terminal100. As shown, the input unit120includes a camera121and a microphone122. The camera121receives and processes image frames of still pictures or video, which are obtained by an image sensor in a video call mode or a photographing mode. In addition, the processed image frames can be displayed on the transparent screen151.

The image frames processed by the camera121can be stored in the memory160or can be externally transmitted via the wireless communication unit110. Optionally, at least two cameras121can be provided to the glass type mobile terminal100according to environment of usage.

An image recognition unit125is a device configured to extract information from an image photographed by the camera121and the image recognition unit125may identify the photographed image. When the photographed image has a character, the image recognition unit converts the character into text data. When the image has a plurality of objects, the image recognition unit extracts an object which will be compared with information stored in the memory160.

When the camera121photographs a motion picture, the motion picture includes still images and the image recognition unit125acquires information of all the still images and the controller processes the acquired information. However, this process might cause unnecessary waste. In addition, if such unnecessary information is extracted and provided to a user, the user using the glass type mobile terminal100wearable like glasses may be inconvenienced.

When the user presses the user input unit128provided in the frame or uses a voice command for acquiring image information, the user can extract appropriate information. Alternatively, when looking at one object for more than a preset time period, the camera121may photograph the same object for more than a preset time period and information on the object may be extracted.

Alternatively, when the user approaches a specific object, the object occupies a large area of the user's view and the image recognition unit125can determine that the user is looking at the corresponding object. Hence, when one object of the images recognized by the camera121occupies a preset range of images composing the motion picture, information on the object may be extracted.

Alternatively, the image recognition unit125may recognize a gesture created by the user and process the input gesture as a user command. For example, the image recognition unit125may recognize a circle drawn by the user as “yes” or perform a function of selecting a character or object located in the circle.

A voice recognition unit126can convert the user's voice input via the microphone1222into text data or recognize the user's voice as a specific command (hereinafter, “a voice command”). The glass type mobile terminal100according to an embodiment of the present invention can include a keyboard. However, a function of the voice recognition unit126configured to recognize the user's voice command is convenient so as to activate the functions of the mobile terminal100.

That is, providing the glass type mobile terminal100with a keyboard may include some disadvantages. However, the glass type mobile terminal100may include a button type user input unit128on the frame and thus a command used frequently can be input via the button type user input unit128. For example, the user can input a command for photographing an image watched by the user or extracting information from the image, or a command for controlling a volume of a sound output from the audio output unit152via the button type user input unit128.

Various types of input units are applicable as the user input unit128and include a module for recognizing a physical pressure from a metallic dome formed therein to generate a signal.

In addition, the sensing unit140senses a state of the mobile terminal100and a user surrounding environment and inFIG. 1includes a gyrosensor141and an acceleration sensor142. The gyrosensor141is a device configured to sense a tilted phase of the mobile terminal100and tilted variation in x, y and z axes, and it can be called as an angular velocity sensor. The gyrosensor senses a rotational movement with respect to each axis to sense tilting and shaking of the mobile terminal case.

A gyrosensor has a spinning top, with three axes and such a gyrosensor is developed to have an improved precision and a compact size such as an optical gyrosensor or a vibration gyrosensor recently, such that a gyrosensor can be mountable in compact-sized electronics including a mobile terminal. In addition, 6-axis sensor realized by MEMS module can be used as the gyrosensor141. The gyrosensor141applied to the present invention is not limited to those types mentioned above and all types of gyrosensors capable of sensing tilting and motion of the glass type mobile terminal100are applicable to the present invention.

The acceleration sensor142can measure a dynamic force, for example, an acceleration of the glass type mobile terminal100and it can detect vibration, shocks and so on from acceleration change. When an object having mass is accelerated, a force is generated and variation of the force is sensed by the acceleration sensor142.

Further, the output unit150generates outputs relevant to the senses of sight, hearing, touch and the like. And, the output unit150includes the transparent screen151, an audio output module152and the like.

The transparent screen151is arranged in front the user's eyes and it is similar to a lens of glasses. The transparent screen arranged in front of the user's eyes allows the user to see objects located in front and it may be transparent and configured to output a text or image to a screen so as to provide the user with information.

As a representative example for the transparent display, there is TOLED (transparent OLED) or the like. A rear configuration of the transparent screen151can be implemented in the optical transmittive type as well. In this configuration, a user can see an object in rear of a terminal body via the area occupied by the transparent screen151of the terminal body.

The memory160can store programs configured for the controller180to process and control and to implement a function for temporarily storing input/output data. The memory160can store new input data therein and extract the stored data. A storage function of the memory160may be expanded by an SD card or an auxiliary storage device.

In addition, the controller180generally controls an overall operation of the mobile terminal100. For example, the controller180may control the wireless communication unit to transmit/receive various signals or process the input data or control the transparent screen151and the audio output unit, only to provide information to the user. The image recognition unit125and the voice recognition unit126mentioned above may be provided in the controller as parts.

The controller180according to an embodiment of the present invention may extract information from the image acquired by the camera121and compare the extracted information with the stored information (S31). Also, the controller180may store the extracted information in the memory160(S32), output associated information extracted from the memory160to output the extracted information to the transparent screen151or search for the associated information on the internet (S34) (seeFIG. 3).

The power supply unit190provides power required by the various components for the glass type mobile terminal100. The power may be internal power, external power, or combinations thereof. Further, a battery may include a built-in rechargeable battery and may be detachably attached to the terminal body for a charging and the like.

Various embodiments described herein may be implemented in a computer-readable medium using, for example, computer software, hardware, or some combination thereof.

Next,FIG. 2illustrates the glass type mobile terminal100according to an embodiment of the present invention including the transparent screen151, the frame101, the camera121and the audio output unit152. The transparent screen151allows the user to see objects located in front, when the user wears the glass type mobile terminal and to provide the user with various information output thereon. The frame101secures the transparent screen151to the user's face to position the transparent screen151in front of their eyes.

In addition, the wireless communication unit110, the input unit120, the sensing unit140, the output unit150and the controller180may be provided in the frame101. The user input unit128may also be provided in the frame101to input a user command thereto or a memory card slot may be provided in the frame101to expand the storage function. The camera121is configured to acquire image information and it may be arranged adjacent to the eyes to acquire images similar to the images acquired by the user's eyes.

As shown in the drawing, one camera121or two or more cameras121for three-dimensional information may be arranged near the eyes. The audio output unit152can also output a loud voice to allow several people at the same time like a speaker or output a sound that is hearable by the user near the ears via a receiver or earphones inserted in the ears, as shown in the drawing.

Next,FIG. 3is a flow chart illustrating a controlling method of a glass type mobile terminal according to one embodiment of the present invention. As shown, the image recognition unit125extracts information from the image recognized by the camera121(S10and S20). The information may be a text data or a specific object. Specific examples of the information will be described later.

The controller180compares the extracted information with stored information (S31) or stores the extracted information in the memory160(S32). Alternatively, the controller180extracts to output associated information stored in the memory160to the transparent screen151(S33) or searches the associated information on the internet (S34).

The comparison between the extracted information and the pre-stored information (S31) means that the extracted information is identified with or similar to the stored information. When the extracted information corresponds to the information stored in the memory160, the extracted information is recognized as corresponding to the stored information. When the extracted information is partially corresponding to the stored information, the extracted information is recognized as associated with the stored information.

When the storing of the extracted information in the memory160, the acquired information may be simply stored. Alternatively, the extracted information is updated in the memory160, when it is the information not repeated after the comparison with the stored information (S31). The newly acquired information may be added to the associated information, when it is associated with the stored information.

In the outputting of the associated information stored in the memory160(S33), the associated information stored in the memory160may be output to the transparent screen151or provided to the user via the audio output unit150, when it the information associated with the acquired information is stored in the memory160. The acquired information may be employed as a search word and information partially corresponding or similar to the acquired information may be searched and the searched information is provided.

In the internet searching for the associated information (S34), the wireless internet module113provided in the wireless communication unit may be used and associated information may be acquired on the internet, when there is no stored information or specific information associated with the extracted information is desired to acquire.

Embodiments of various function performed by the glass type mobile terminal100according to an embodiment of the present invention mentioned in reference toFIG. 3will now be described.

In particular,FIGS. 4 to 7are diagrams illustrating an operational state of the glass type mobile terminal according to one embodiment of the present invention. In this embodiment, the image acquired by the camera121includes a calendar310as shown inFIG. 4.

When the user looks at the calendar310for more than a preset time period or in a close distance, information is extracted from the calendar310. When one of the conditions or both of the conditions is or are satisfied, the information may be set to be extracted from the calendar310. Alternatively, the acquiring of the information may be implemented by the user pressing the user input unit128attached to the glass type mobile terminal100or a voice command.

When the image recognition unit125recognizes the calendar310, a marker511may be displayed on an area recognized as the calendar310as shown inFIG. 4. The information that can be extracted from the calendar310may correspond to the month311, the year312and the date313. Information on the year, month and date of the calendar may be searched through the memory160. In other words, the user's scheduler may be searched and information stored in the scheduler by the user is searched in the memory160.

In addition, searching for the stored information through the memory160may be implemented automatically. Alternatively, as shown inFIG. 4, the image recognition unit125may recognize the calendar310and extract information from the image. After that, buttons512aand512bmay be output to the transparent screen151in a lower area of the transparent screen151to confirm whether the user desires to display the stored information. The buttons512aand512bare not physically touched but rather the user can input a voice command corresponding to the corresponding button or use the user input unit128.

As shown inFIG. 5, the information513searched in the memory160may be output to the transparent screen151, and overlapped with the calendar310the user is looking at. When the user moves, the size or position of the calendar310moves in the user's view according to the user's movement and the size or position of the stored information513output to the transparent screen151may be changed.

As shown inFIG. 6, when the size of the calendar310is increased, a space of the transparent screen151where the information is output can also be increased such that the stored information513′ can be displayed more specifically. When the user desires to acquire contents stored in the memory160and information associated with a specific data or period, information can be searched on an external internet via the wireless communication unit110.

In addition,FIG. 5illustrates that information514associated with Chuseok, Korean Thanksgiving Day is additionally searched and output to the transparent screen151. The internet search may be implemented by the user's voice command or the user's input to the user input unit128or a gesture created by the user is read and recognized as a command by the image recognition unit125.

According to another embodiment, when the user speaks “search information “a concert on October 13”, the voice recognition unit126recognizes contents of the voice as a voice command, and the controller180searches information on the concert on October 13 via the wireless internet module113according to the command.

The information that can be extracted from the calendar310can be recognized from characters315written on the corresponding calendar310, rather than the year, the month and the date311,312and313, to be stored as text data. In this example, it is checked whether there is overlap between the extracted information and the information stored in the memory160and the content not overlapped may be newly added. The storing may be performed by activating the button output to the transparent screen151by a voice command or the information of the areas515and516selected by the user may be stored as shown inFIG. 7.

Next,FIGS. 8 to 12are diagrams illustrating an operational state of a glass type mobile terminal according to another embodiment of the present invention. In this embodiment, the image photographed by the camera121is a memo320as shown inFIG. 8and the image recognition unit125converts a character written on the memo into text data via OCR. The stored images result in a large data capacity and it is difficult to edit the images, and thus the hand writing on the memo is converted into the text data.

The extracted text data is compared with the information stored in the memory160and associated information is searched. For example, conference minutes or notes taken in a class may be made in connection with the contents stored in the memory160and the information is retrieved so as to identify the contents recorded in the last conference or class.

To extract the associated information, the extracted information is compared with the stored information as mentioned above and information corresponding to the extracted text data is extracted. InFIG. 8, characters321are recognized from the image of the memo320. For example, text such as “conference minutes” and “smart glasses” related to a theme are extracted and searched for stored information. Even in this embodiment, the character recognized as an object for information extraction may be displayed with a marker521.

Once related information is retrieved from the memory, related information522aand522bmay be displayed on the transparent screen151as shown inFIG. 8. All of the contents possessed by the related information do not have to be displayed and only a name or date can be displayed in a thumbnail mode. When selecting the corresponding memo, the user may select whether to check contents of the memo523a, to store a new memo grouped together with the selected memo523b, or to store a memo following the selected memo523c, as shown inFIG. 9.

Once the user starts to make a memo, the image recognition unit125can recognize characters newly written by the user in real time and convert the recognized characters into a digital text. Further, the converted text may be displayed524on the transparent screen151as shown inFIG. 10, to allow the user to check whether the text conversion is performed appropriately.

When wanting to search a specific content of the memo on the internet, the user may select the specific content and search the selected content via the wireless internet module113. Alternatively, the searching may be performed by a voice command. However, sometimes it is difficult to use the voice command when the memo is taken in a library or during a conference.

Thus, in the present invention, when a gesture323configured to specify a searching content321is made by the user, the image recognition unit125may recognize the gesture323as a user command. When wanting to check details of the searched content thumbnail525displayed on the transparent screen151, an enlarge button525amay be activated to check the searched content in a large screen.

A memo pad generally has a limited size. As shown inFIG. 12, when the user starts a new page, the image recognition unit125may recognize the page replacement and ask the user whether the searched content should be stored as a new page526a, to store it as a new text526bor to store it in one page, following the former page.

Next,FIGS. 13 to 18are diagrams illustrating an operational state of a glass type mobile terminal according to another embodiment of the present invention. In this embodiment, the motion picture photographed by the camera121includes an object330′, and the image recognition unit125extracts and process an image of the corresponding object.

The user may record the image of the object330′ in the memory160as stored information. For example, as shown inFIG. 13, images of objects330′ having the danger of being lost such as a wallet, a watch and a key are stored. Preferably, a plurality of images photographed in as many directions as possible are stored. Such object having the images stored in the memory can be referenced to as “a designated object”.

As shown inFIG. 14, an image of the object330is extracted from the motion picture photographed by the camera121, while being distinguished from the background, and the extracted image is compared with the image of the designated object330′ stored in the memory160to check whether the extracted image corresponds to the image of the designated object330′.

When the extracted image of the object330corresponds to the image of the designated image330′ based on the result of the comparison, the time and the place of the motion picture photographed by the camera121is stored (seeFIG. 15). In other words, a log record file of the designated object330′ is generated.

If the user loses the designated object330′, the log record of the corresponding object is searched and the last place where the designated object330′ is seen and photographed can be provided to make it easy for the user to find the designated object330′.

Alternatively, when the image of the object330extracted from the motion picture photographed by the camera121is identified with the image of the designated object330′, information related with the designated object330′ can be output. The stored information may be output on the transparent screen151as a text or to the audio output unit152as a voice.

For example, when the user is on a diet as shown inFIG. 16, high calorie foods including meat and a cake may be set as designated objects330′. When the image of the extracted object330is identified with the designated objects330′, a message telling the user not to eat the object330may be output.

Alternatively, when a conference room is set as a designated object330′ as shown inFIG. 17, information534on a conference schedule may be extracted from the memory160and output. Alternatively, when a specific person's face is set as the designated object330′ as shown inFIG. 18, a memo533related with a specific person may be searched from the memory160and output.

When the designated object330′ is a specific object as shown inFIG. 14 or 17, a high match rate between the image of the designated object330′ and the extracted image of the object330makes the object330corresponding to the designated object330′. When the designated object330′ is an object as shown inFIG. 16, a low match rate between the images may make the object330set to be corresponding to the designated object330′.

When the extracted image of the object includes time information as shown inFIG. 17, only information close to the corresponding time may be provided or information on the time several hours before the time may be provided.

Next,FIGS. 19 to 22are diagrams illustrating an operational state of the glass type mobile terminal according to still another embodiment of the present invention. In this embodiment, the image is money341such as a bill and a coin as shown inFIG. 20and the image recognition unit125extracts the amount541of the money.

When a sales clerk says the amount of money the user has to pay as shown inFIG. 19, the voice recognition unit including the microphone122extracts the amount the user has to pay as digital information. The amount of money the user has to pay may be the voice of the clerk or an image including the amount, such that the image recognition unit125may extract the amount of the money the user has to pay from the image.

The amount of the money541is extracted from the image of the money341the user takes outs and gives to the clerk as digital information as shown inFIG. 20. The amount of the money the user has to pay is compared with the amount541of the money the user took out, such that the change542may be calculated based on the result of the comparison.

Next, the image photographed by the camera121is a receipt345as shown inFIG. 21and the information extracted from the receipt345may include the amount345aand the date345brecorded on the receipt345. In addition, information on the place345cor a credit/debit card345dused may be extracted from the image of the receipt345. The controller180can then add the extracted information345a,345b,345cand345dto account book stored in the memory160. In this embodiment, the user can conveniently maintain their accounts, without having to manually input the information.

As described above, the images acquired according to the user's view may be processed and various types of information may be extracted, such that the extracted information may be input or compared with the stored information or such that the stored information may be searched to provide the searched information to the user. Accordingly, new information can be acquired easily and utilized.

Furthermore, the information related to an object that could be easily lost or the corresponding object330may be provided to the user to assist the user's power of memory.

The present invention encompasses various modifications to each of the examples and embodiments discussed herein. According to the invention, one or more features described above in one embodiment or example can be equally applied to another embodiment or example described above. The features of one or more embodiments or examples described above can be combined into each of the embodiments or examples described above. Any full or partial combination of one or more embodiment or examples of the invention is also part of the invention.