Source: https://patents.google.com/patent/JP2017143409A/en
Timestamp: 2020-01-24 04:53:58
Document Index: 55235534

Matched Legal Cases: ['art, 22', 'art, 24', 'art, 26', 'art, 28', 'art, 30', 'art, 32', 'art, 100']

JP2017143409A - Mobile terminal device and program - Google Patents
Mobile terminal device and program Download PDF
JP2017143409A
JP2017143409A JP2016023407A JP2016023407A JP2017143409A JP 2017143409 A JP2017143409 A JP 2017143409A JP 2016023407 A JP2016023407 A JP 2016023407A JP 2016023407 A JP2016023407 A JP 2016023407A JP 2017143409 A JP2017143409 A JP 2017143409A
JP2016023407A
章倫 鈴木
Akimichi Suzuki
2016-02-10 Application filed by 株式会社Ｊｖｃケンウッド, Jvc Kenwood Corp filed Critical 株式会社Ｊｖｃケンウッド
2016-02-10 Priority to JP2016023407A priority Critical patent/JP2017143409A/en
2017-08-17 Publication of JP2017143409A publication Critical patent/JP2017143409A/en
PROBLEM TO BE SOLVED: To provide a technique that displays information depending on a surrounding environment in an easy-to-understand manner.SOLUTION: A positioning unit 20 determines first positional information on a mobile terminal device 10. A communication unit 22 obtains second positional information determined by another mobile terminal device 10. An imaging unit 26 photographs a surrounding environment. An acquisition unit 24 acquires information on a state depending on the environment around the other mobile terminal device 10. A display unit 32 displays another device icon indicating another measuring unit 20 and also displays the acquired information by associating it with the other device icon on the taken image.SELECTED DRAWING: Figure 2
The present invention relates to display technology, and more particularly, to a mobile terminal device and a program for displaying captured images.
Augmented reality (AR) technology displays a variety of information on a captured image. For example, the positions of other communication terminals existing around the communication terminal are displayed on the landscape image using AR technology. Furthermore, the attribute of the user who uses another communication terminal and the information regarding the compatibility between users are also displayed (for example, patent document 1).
JP 2012-221330 A
Since information regarding users who are using other communication terminals is displayed on the landscape image, it is easy to see what kind of users exist around the communication terminals. However, sufficient information regarding the availability of communication between communication terminals and communication quality, that is, information depending on the environment around the communication terminal is not displayed.
The present invention has been made in view of such circumstances, and an object of the present invention is to provide a technique for easily displaying information depending on the surrounding environment.
In order to solve the above problems, a mobile terminal device according to an aspect of the present invention is a mobile terminal device, and communicates with a positioning unit that measures the first position information of the mobile terminal device and another mobile terminal device. Thus, the communication unit that acquires the second position information measured in the other mobile terminal device, the imaging unit that images the environment around the mobile terminal device, and the environment around the other mobile terminal device In addition to the coordinates derived based on the acquisition unit that acquires information about the state, the first position information measured by the positioning unit, and the second position information acquired by the communication unit on the image captured by the imaging unit A display unit that displays the other device icon indicating the mobile terminal device and displays the information acquired by the acquisition unit in association with the other device icon.
According to the present invention, information depending on the surrounding environment can be displayed in an easy-to-understand manner.
It is a figure which shows the structure of the communication system which concerns on Example 1 of this invention. It is a figure which shows the structure of the mobile terminal apparatus of FIG. It is a figure which shows the image produced | generated in the process part of FIG. It is a figure which shows another image produced | generated in the process part of FIG. It is a sequence diagram which shows the display procedure by the communication system of FIG. It is a figure which shows the image produced | generated in the process part which concerns on Example 2 of this invention. It is a figure which shows another image produced | generated in the process part which concerns on Example 2 of this invention. It is a figure which shows another image produced | generated in the process part which concerns on Example 2 of this invention. It is a sequence diagram which shows the display procedure by the communication system which concerns on Example 2 of this invention. It is a sequence diagram which shows another display procedure by the communication system which concerns on Example 2 of this invention. It is a flowchart which shows the display procedure by the process part of FIG. It is a figure which shows the data structure of the database memorize | stored in the memory | storage part which concerns on Example 3 of this invention. It is a figure which shows the image produced | generated in Example 3 of this invention. It is a figure which shows the image produced | generated in Example 4 of this invention.
Before describing the present invention specifically, an outline will be given first. Embodiment 1 of the present invention relates to a communication system including a plurality of mobile terminal apparatuses. This communication system is compatible with, for example, a commercial radio system, and a plurality of mobile terminal apparatuses communicate directly with each other without going through a base station apparatus. Here, the communication form may be one-to-one or one-to-many, and the latter corresponds to group communication. The mobile terminal device includes an imaging device, can display augmented reality, and can acquire GPS (Global Positioning System) information. Such a mobile terminal device may be configured as a single device or a combination of two or more devices, but is assumed to be the former for the sake of clarity.
In the latter case, for example, it is realized by a combination of a wireless device compatible with a business wireless system and a smartphone not compatible with a business wireless communication system. The wireless device and the smartphone can communicate with each other by a communication system different from the business wireless system, and the wireless device relays communication between the business wireless system and the smartphone. Furthermore, the smartphone may be any device that includes an imaging device, can display augmented reality, and can acquire GPS (Global Positioning System) information, and other devices may be used instead of the smartphone.
Each mobile terminal apparatus measures the current position by the GPS function, and exchanges position information with each other by the business wireless system. Based on the exchanged position information, the mobile terminal device displays an icon indicating the other mobile terminal device (hereinafter referred to as “other device icon”) on the captured image of the surrounding environment. In addition, the mobile terminal device measures the signal strength (signal strength related to the received radio wave) when receiving a signal from another mobile terminal device as information relating to a state depending on the surrounding environment of the other mobile terminal device. The signal strength is displayed together with other device icons. In the image displayed in this way, the position where the other mobile terminal device exists is shown on the image of the surrounding environment where the other terminal exists, and the signal strength is also shown. Therefore, the user of the mobile terminal device is provided with an image displayed as information on a state depending on the surrounding environment, for example, the signal strength is lowered because there is a building in between, and is shown in the image It is possible to grasp the ease of communication with other mobile terminal devices in the environment.
FIG. 1 shows a configuration of a communication system 100 according to Embodiment 1 of the present invention. The communication system 100 includes a first mobile terminal device 10a, a second mobile terminal device 10b, a third mobile terminal device 10c, and a fourth mobile terminal device 10d, which are collectively referred to as the mobile terminal device 10. Here, the number of mobile terminal apparatuses 10 included in the communication system 100 is not limited to “4”, and may be larger or smaller. As described above, the plurality of mobile terminal apparatuses 10 are compatible with the business wireless system, and perform voice communication and data communication. Further, as a communication form, one-to-one communication such as transmission from the first mobile terminal device 10a to the second mobile terminal device 10b is executed, or the first mobile terminal device 10a to the second mobile terminal device 10b to fourth. One-to-many communication such as transmission to the mobile terminal device 10d is executed.
Here, the plurality of mobile terminal apparatuses 10 directly communicate with each other without going through the base station apparatus. Therefore, they must be within a range where they can communicate directly with each other. However, even when the two mobile terminal devices 10 are close to each other, the quality of communication between them may not be good due to the presence of obstacles and topography. On the other hand, each of the plurality of mobile terminal devices 10 is carried by a user and is movable. Therefore, for example, a user who uses the first mobile terminal device 10a may move to a position where the communication is good when the communication with the second mobile terminal device 10b is not good. In order to realize this, it is desirable that information regarding a state depending on the environment around the other mobile terminal device 10 is presented to the user in an easily understandable manner.
Such presentation is realized by, for example, an application program executed in the mobile terminal device 10. In the following, a process for realizing such presentation will be described. To simplify the description, the state of the second mobile terminal device 10b and the like around the first mobile terminal device 10a is changed to the first mobile terminal. Assume a case where it is presented to the user of the device 10a. Such processing is similarly executed in the other mobile terminal apparatuses 10.
FIG. 2 shows the configuration of the mobile terminal apparatus 10. The mobile terminal device 10 includes a positioning unit 20, a communication unit 22, an acquisition unit 24, an imaging unit 26, a processing unit 28, a storage unit 30, and a display unit 32. This configuration is common to the first mobile terminal device 10a to the fourth mobile terminal device 10d in FIG.
The positioning unit 20 measures the position where the mobile terminal device 10 exists by receiving a signal from a GPS satellite (not shown). The position is indicated by, for example, latitude, longitude, and altitude. Since a known technique may be used for positioning by GPS, description thereof is omitted here, but a moving speed and a moving direction may be detected in addition to the position. In addition, the positioning unit 20 has an electronic compass function, and measures the current direction. The positioning unit 20 outputs these pieces of information (hereinafter referred to as “first position information”) to the processing unit 28.
The communication unit 22 corresponds to, for example, a commercial wireless system and communicates with other mobile terminal devices 10. The communication unit 22 receives the first position information from the processing unit 28 and transmits a signal including the first position information to another mobile terminal device 10. On the other hand, the communication unit 22 receives signals from other mobile terminal devices 10. This signal includes information measured by another mobile terminal device 10 (hereinafter referred to as “second position information”). The communication unit 22 outputs the second position information to the processing unit 28.
The imaging unit 26 images the environment around the mobile terminal device 10 and generates an image. The surrounding environment corresponds to the scenery around the mobile terminal device 10. Here, the image may be a still image or a moving image. The imaging unit 26 outputs the captured image to the processing unit 28. The output image may be digital data, but an image as digital data is also called an “image”.
The acquisition unit 24 acquires information related to a state depending on the environment around the other mobile terminal device 10. In particular, the acquisition unit 24 measures the signal strength when the communication unit 22 receives a signal from another mobile terminal device 10 as information relating to a state depending on the environment around the other mobile terminal device 10. This signal may include the above-described second position information, but may not include it. The acquisition unit 24 processes a combination of the measured signal strength and information of another mobile terminal device 10 that has transmitted a signal for measuring the signal strength (hereinafter also referred to as “signal strength”) 28. Output to.
Note that the signal strength may be transmitted from the communication unit 22 to another mobile terminal device 10. In that case, a signal received by the communication unit 22 and also a signal from another mobile terminal device 10 is measured at the other mobile terminal device 10 based on the signal transmitted by the mobile terminal device 10. Signal strength is included. The acquisition unit 24 may use the signal strength included in the signal included in the communication unit 22 as information regarding a state depending on the surrounding environment of the other mobile terminal device 10. That is, as information regarding the state depending on the environment around the other mobile terminal device 10, the signal strength when the communication unit 22 receives a signal from the other mobile terminal device 10, or the mobile terminal device that performs image display The signal strength when 10 is transmitted and received by another mobile terminal apparatus 10 can be used.
The processing unit 28 inputs the first position information from the positioning unit 20, the second position information from the communication unit 22, the signal intensity from the acquisition unit 24, and the image from the imaging unit 26. The processing unit 28 arranges the other device icon indicating the other mobile terminal device 10 on the expansion space in the image by executing the AR technique. Specifically, the processing unit 28 associates the coordinates in the image from the imaging unit 26 with the latitude / longitude (position information) based on the current position in the first position information and the direction information. Put it on.
Here, if the angle of view in the image is known, when an arbitrary point in the image is fixed to a predetermined latitude / longitude, the latitude / longitude of another point is uniquely derived. For example, when the coordinates of the first point on the image are (x1, y1) and the coordinates of the second point different from the first point are (x2, y2), the latitude / longitude for the first point is If determined, the latitude and longitude for the second point are also determined. The processing unit 28 derives the latitude and longitude of the second point by associating the current position in the first position information with the first point, and executes such processing on various coordinates on the image. By doing so, the coordinates in the image are associated with the latitude and longitude.
In addition, the processing unit 28 converts the second position information from the communication unit 22 into coordinates using the correspondence between the coordinates in the image and the latitude / longitude. Further, the processing unit 28 arranges other device icons at the converted coordinates on the image. FIG. 3 shows an image 50 generated by the processing unit 28. The image 50 shows a landscape that is the environment around the first mobile terminal device 10a, and on the landscape, a second other device icon 200b, a third other device icon 200c, which are collectively referred to as another device icon 200, A fourth other device icon 200d is shown. The second other device icon 200b is an icon for indicating the second mobile terminal device 10b, the third other device icon 200c is an icon for indicating the third mobile terminal device 10c, and a fourth other device icon 200d. Is an icon for indicating the fourth mobile terminal apparatus 10d. Therefore, the second other device icon 200b is arranged at coordinates corresponding to the latitude / longitude in the second position information from the second mobile terminal device 10b. The same applies to the third other device icon 200c and the fourth other device icon 200d. Since the other description will be described later, the description returns to FIG.
The processing unit 28 arranges the signal strength information acquired by the acquisition unit 24 on the image 50 so as to be associated with each other device icon 200. Further, the processing unit 28 uses the lower center portion of the image 50 as the position of the first mobile terminal device 10a to indicate that communication with each other device icon 200 is possible (hereinafter referred to as “transmission line line”). "). At that time, the processing unit 28 changes the display mode of the transmission line according to the value of the signal strength. The display mode includes color shading, color difference, line type, line thickness, and the like. For example, the processing unit 28 darkens the color or thickens the line as the signal intensity increases. As described above, the processing unit 28 changes the display mode of the signal strength according to the value of the signal strength measured by the acquisition unit 24.
In the image 50 shown in FIG. 3, the second other device information 202b is arranged so as to be associated with the second other device icon 200b, and the second other device information 202b corresponds to the second mobile terminal device 10b. Indicates the signal strength. A second transmission line 204b is arranged between the second other device icon 200b. Such a display is the same for the third other device icon 200c and the fourth other device icon 200d. The second other device information 202b, the third other device information 202c, and the fourth other device information 202d are collectively referred to as the other device information 202, and the second transmission line 204a, the third transmission line 204c, and the fourth transmission line. The line 204d is collectively referred to as the transmission line 204.
As described above, the transmission line 204 is shown while changing the type of line (solid line / dotted line) and the color of the line (dark color / single color or color difference). It is also possible to assign a different color to each other device icon 200. For example, the second other device icon 200b that is relatively far away and located behind the building has a slightly lower received signal strength, and therefore the second transmission line 204b is a thin broken line with a narrow interval. Expressed. In addition, since the third other device icon 200c that is far away and completely located behind the building has a low received signal strength, the third transmission line 204c is a broken line with a thin color interval. It is expressed by When the signal strength is low, the user of the first mobile terminal device 10a can move in that direction because he or she can determine that the communication partner is not behind the building or the like by looking at the displayed image 50. This makes it possible to easily improve communication quality. On the other hand, the fourth other device icon 200d arranged at a short distance and a place with a good line of sight has a high received signal strength, so the fourth transmission line 204d is represented by a dark solid line. Returning to FIG.
In addition, when the information from the other mobile terminal apparatus 10 includes information on the state depending on the environment around the other mobile terminal apparatus 10, the acquisition unit 24 determines whether the signal received by the communication unit 22 is Such information (hereinafter referred to as “state information”) may be acquired. The acquisition unit 24 outputs the state information to the processing unit 28, and the processing unit 28 arranges the state information in association with the other device icon 200 on the image 50. FIG. 4 shows another image 50 generated by the processing unit 28. Here, the second other device information 202b is arranged in association with the second other device icon 200b. In the second other device information 202b, the name / state / location / speed / direction / position information update date and time can be displayed, unlike the signal strength described above. If the displayed second other device icon 200b is selected by a tap or the like, a second submenu 206b is displayed in association with the second other device icon 200b. In the second submenu 206b, a communication instruction such as a call / message / transmission is displayed. For example, when “call” is selected from the second submenu 206b, a call is made with the second mobile terminal device 10b. Returning to FIG.
The storage unit 30 stores various information used for processing. For example, the first position information from the positioning unit 20, the second position information from the communication unit 22, the signal intensity from the acquisition unit 24, and the image from the imaging unit 26 are stored. The display unit 32 displays the image 50 generated by the processing unit 28. For example, the image 50 shown in FIGS. 3 and 4 is displayed. The display unit 32 may be a touch panel.
The operation of the communication system 100 configured as above will be described. FIG. 5 is a sequence diagram showing a display procedure by the communication system 100. The second mobile terminal apparatus 10b performs positioning (S10), and transmits a signal including the second position information to the first mobile terminal apparatus 10a (S12). The first mobile terminal apparatus 10a receives the signal and measures the signal strength (S14). The third mobile terminal apparatus 10c performs positioning (S16), and transmits a signal including the second position information to the first mobile terminal apparatus 10a (S18). The first mobile terminal apparatus 10a receives the signal and measures the signal strength (S20). The fourth mobile terminal apparatus 10d performs positioning (S22), and transmits a signal including the second position information to the first mobile terminal apparatus 10a (S24). The first mobile terminal apparatus 10a receives the signal and measures the signal strength (S26). The first mobile terminal device 10a displays the image 50 (S28).
According to the present embodiment, the other device icon is displayed on the image, and information related to the state depending on the surrounding environment of the other mobile terminal device is also displayed, so that the information depending on the surrounding environment can be displayed in an easy-to-understand manner. . In addition, since the other device icon is displayed at the coordinates arranged at the position of the other mobile terminal device, the position of the other mobile terminal device can be easily grasped only by looking at the image. Further, since the signal strength is displayed as information related to the state depending on the environment around the other mobile terminal devices, it is possible to associate the communication quality with other mobile terminal devices. Further, since the display mode is changed according to the signal strength, the signal strength can be displayed in an easily understandable manner.
Further, since the signal strength is displayed, it is possible to indicate a position where the communication environment is good. In addition, since the other device icon and the signal strength are displayed, it can be confirmed from the augmented reality distance and the signal strength whether or not communication with another mobile terminal device is possible. Further, since the other device icon and the signal intensity are displayed, the coverage area can be confirmed in augmented reality, and the movement can be promoted when communication is interrupted. In addition, since the other device icon and the signal strength are displayed, it is possible to promptly take an appropriate action by causing each user to grasp the situation when a problem occurs and to take an action independently. Moreover, since information is displayed on augmented reality, the situation can be intuitively grasped.
In the present embodiment, the case where the signal strength is used as the information related to the state depending on the environment around the mobile terminal device has been described as an example, but other types of information may be used. For example, a call microphone of another mobile terminal device picks up some noise sounds (wind noise, traffic noise, etc.) depending on the environment around the other mobile terminal devices in addition to the voice of the user. Therefore, in another mobile terminal device, the output signal of the microphone is analyzed, the signal intensity of the noise (ambient environmental sound) component is measured, and the information regarding the signal strength depending on the surrounding environment of the other mobile terminal device It can be. Then, the signal strength of the noise component is transmitted from another mobile terminal device to the mobile terminal device (first mobile terminal device 10a) that performs image display, and instead of the signal strength of the received radio wave shown in FIG. The signal strength may be displayed. Alternatively, an audio signal transmitted from another mobile terminal device may be analyzed in a mobile terminal device (first mobile terminal device 10a) that performs image display, and the signal strength of the noise component may be calculated. Alternatively, the intensity of the noise component may be subtracted from the intensity of the entire acoustic signal to calculate the intensity of the component related to the human voice that is originally necessary, and display based on the calculated intensity may be performed. The user of the first mobile terminal device 10a can instruct the call partner to move to a position where acoustic noise is likely to be reduced while looking at the display screen when it is difficult to hear the call partner's voice. By looking at the display screen, for example, it is possible to identify the shadow of a building that is not directly exposed to the wind or the position where traffic noise is likely to be low. For this reason, call quality can be improved.
In addition, when the mobile terminal device has a function that depends on ambient illuminance (for example, a solar cell charging function), the illuminance measured by the other mobile terminal device depends on the surrounding environment of the other mobile terminal device. It is good also as information about. Alternatively, the power generation amount generated by the ambient light per unit time, the charge amount of the solar battery, and the like may be information relating to a state depending on the surrounding environment of another mobile terminal device. Then, instead of the signal intensity of the received radio wave shown in FIG. 3, the amount of power generated by ambient light may be displayed. The user of the mobile terminal device (first mobile terminal device 10a) that performs image display moves another mobile terminal device to a position where the efficiency of the function depending on the surrounding illuminance is likely to increase while looking at the display screen. Can do.
Next, Example 2 will be described. Similar to the first embodiment, the mobile terminal device according to the second embodiment displays the other device icon indicating another mobile terminal device on the captured image by using the AR technology. Also in the second embodiment, information regarding a state depending on the environment around other mobile terminal devices is also displayed on the image, but this information is different from the first embodiment. In the second embodiment, status information indicating that an emergency state is received is received from another mobile terminal device, and display is performed so that the emergency status can be recognized. The communication system 100 and the mobile terminal device 10 according to the second embodiment are of the same type as those shown in FIGS. Here, it demonstrates centering on the difference with Example 1. FIG.
The communication unit 22 in FIG. 2 receives a signal transmitted from another mobile terminal device 10. The received signal includes state information from another mobile terminal apparatus 10. The acquisition unit 24 acquires the state information received by the communication unit 22. Here, the state information corresponds to information regarding a state depending on the environment around the other mobile terminal device 10. In the state information, emergency information is indicated. The emergency information is information that is output as an emergency notification from another mobile terminal device 10, and is classified into “EMERGENCY information” and “EMERGENCY MAN-DOWN information”. Although both are information indicating an emergency or emergency, “EMERGENCY information” is output when a user of another mobile terminal apparatus 10 performs a predetermined operation, whereas “EMERGENCY MAN-DOWN information” is output. "Is information that is automatically output when a user carrying another mobile terminal device 10 falls. For example, when an explosion accident occurs at a certain point and the user of the mobile terminal device 10 that is nearby falls due to the impact, “EMERGENCY MAN-DOWN information” is output. In such a case, the information regarding the state “EMERGENCY MAN-DOWN” can be said to be information determined by the surrounding environment such as an explosion accident. Regarding the output of emergency information by other mobile terminal apparatuses 10, any known technique may be used, and thus description thereof is omitted here. The acquisition unit 24 outputs emergency information to the processing unit 28.
The processing unit 28 also inputs emergency information that is state information from the acquisition unit 24. The processing unit 28 generates the image 50 by arranging the input emergency information in association with the other device icon 200. FIG. 6 shows an image 50 generated by the processing unit 28 according to the second embodiment of the present invention. The image 50 is shown as in FIG. Here, it is assumed that emergency information is acquired from the second mobile terminal device 10b and the third mobile terminal device 10c. The processing unit 28 displays the second other device icon 200b in FIG. 6 in a display mode different from the second other device icon 200b in FIG. 3 in order to arrange emergency information in association with the other device icon 200. That is, the other device icon 200 for the other mobile terminal device 10 that has received the emergency information is changed from the other device icon 200 for the other mobile terminal device 10 in the normal state so as to be conspicuous.
Further, it is also assumed that the emergency information from the second mobile terminal apparatus 10b is “EMERGENCY information” and the emergency information from the third mobile terminal apparatus 10c is “EMERGENCY MAN-DOWN information”. Therefore, the third other device icon 200c is an icon having a shape that the user has fallen to the ground, and has a display mode different from that of the second other device icon 200b. Further, second state information 208b is arranged in association with the second other apparatus icon 200b, and third state information 208c is arranged in association with the third other apparatus icon 200c. The second state information 208b indicates “EMERGENCY information”, and the third state information 208c indicates “EMERGENCY MAN-DOWN information”.
When the emergency level (severity level) of emergency information is divided into a plurality of levels, the color and shape of the other device icon 200 may be changed according to the level. As a display method, there is a method of constantly displaying / changing / flashing the other device icon 200 according to emergency information, changing / flashing the background color / displaying large / non-displaying, etc. And combinations are possible. Moreover, you may make it emit a notification sound. The emergency information may be displayed on all the mobile terminal devices 10 that have received the emergency information, or only on the mobile terminal devices 10 in the same group as the mobile terminal device 10 that has transmitted the emergency information. Good. Returning to FIG.
The image 50 generated in the processing unit 28 may be different from that in FIG. As described above, when the EMERGENCY MAN-DOWN information is transmitted, the user of the mobile terminal device 10 that is the transmission source has fallen and is often in a low position. Therefore, the other device icon 200 of the mobile terminal device 10 that has transmitted the EMERGENCY MAN-DOWN information is hidden behind another object or building and is not placed in the image 50. In order to cope with this, when the third mobile terminal apparatus 10c transmits the EMERGENCY MAN-DOWN information, if the user of the third mobile terminal apparatus 10c can be imaged by the imaging unit 26 of the second mobile terminal apparatus 10b, 2. Imaging is performed in the mobile terminal device 10b. The second mobile terminal device 10b transmits an image (hereinafter referred to as “photo image”) obtained by imaging the user of the third mobile terminal device 10c to the first mobile terminal device 10a and the like, and a communication unit of the first mobile terminal device 10a 22 also receives photographic images. The acquisition unit 24 acquires the photographic image received by the communication unit 22 and outputs it to the processing unit 28.
The processing unit 28 generates the same image 50 as in FIG. 6, but at this time, the photographic image acquired from the acquisition unit 24 is synthesized. FIG. 7 shows another image 50 generated in the processing unit 28 according to the second embodiment of the present invention. Here, the third other device icon 200c is not arranged because it is hidden behind the building. Instead, a photographic image 210 taken by the second mobile terminal device 10b is shown. Note that the second mobile terminal device 10b receives the position information of the third mobile terminal device 10c, the position and direction information of the second mobile terminal device 10b, and the angle of view information of the imaging unit 26 of the second mobile terminal device 10b. Are used to create an image (an image similar to that shown in FIG. 6) in which the third mobile terminal device 10c is arranged in the expansion space viewed from the second mobile terminal device 10b, and transmit it to the first mobile terminal device 10a. May be. At that time, drawing of the transmission line 204 is omitted. Returning to FIG.
When the other device icon 200 of the mobile terminal device 10 that has transmitted the EMERGENCY MAN-DOWN information as described above is not arranged in the image 50, the processing unit 28 uses the first position information and the second position information as the basis. The other device icon 200 may be arranged at a different coordinate different from the coordinate derived in (1). At this time, the processing unit 28 also arranges information (hereinafter referred to as “coordinate information”) related to the relationship between different coordinates in association with the other device icon 200. That is, when the other device icon 200 is arranged at a position different from the position where the other mobile terminal device 10 actually exists, the other mobile terminal device 10 is actually used with reference to the display position of the other device icon 200. The relative positional relationship of the positions existing in is arranged as character information.
FIG. 8 shows still another image 50 generated by the processing unit 28 according to the second embodiment of the present invention. For example, when the third mobile terminal device 10c is located at latitude X, longitude Y, and altitude Z (units are all [m]), the processing unit 28 sets the third other device icon at a position where the altitude is 10 m higher (Z + 10 m). 200c is arranged. In addition, the processing unit 28 arranges coordinate information 212 in which characters such as “this 10 m below” are indicated in the vicinity of the third other device icon 200c. If it does in this way, even when the user of other mobile terminal devices 10 is falling behind the building, the position is easy to be specified.
The operation of the communication system 100 configured as above will be described. FIG. 9 is a sequence diagram illustrating a display procedure by the communication system 100 according to the second embodiment of the present invention. The second mobile terminal device 10b detects emergency (S100). The second mobile terminal apparatus 10b transmits state information indicating the EMERGENCY information to the first mobile terminal apparatus 10a (S102), and the first mobile terminal apparatus 10a acquires the EMERGENCY information (S104). The third mobile terminal device 10c detects emergency man down (S106). The third mobile terminal apparatus 10c transmits state information indicating the EMERGENCY MAN-DOWN information to the first mobile terminal apparatus 10a (S108), and the first mobile terminal apparatus 10a acquires the EMERGENCY MAN-DOWN information ( S110). The first mobile terminal device 10a displays the image 50 (S112).
FIG. 10 is a sequence diagram illustrating another display procedure by the communication system 100 according to the second embodiment of the present invention. The second mobile terminal device 10b captures a photographic image of the user carrying the third mobile terminal device 10c (S150). The second mobile terminal device 10b transmits a photographic image to the first mobile terminal device 10a (S152). The first mobile terminal device 10a displays the image 50 obtained by combining the photographic images (S154).
FIG. 11 is a flowchart showing a display procedure by the processing unit 28. If the position of the other mobile terminal apparatus 10 that has acquired the EMERGENCY MAN-DOWN information is not displayable in the image 50 (N in S200), the processing unit 28 actually uses the position of the mobile terminal apparatus 10 based on the position of the mobile terminal apparatus 10. Position information that can be displayed in the image 50 is created by performing processing such as raising the altitude from the position of (S202). If the position of another mobile terminal device 10 that has acquired EMERGENCY MAN-DOWN information can be displayed in the image 50 (Y in S200), step 202 is skipped. The storage unit 30 displays the image 50. That is, when the state of the mobile terminal apparatus 10 satisfies a predetermined condition, the position (actual position) of the mobile terminal apparatus 10 is changed to a position (position for display) that can be displayed in the extended space of the display screen. You may display above. At this time, information regarding the difference (relationship) between the actual position and the position for display may be displayed.
According to the present embodiment, the status information from the other mobile terminal devices is displayed in association with the other device icons as the information related to the status depending on the surrounding environment of the other mobile terminal devices. Information can be displayed in an easy-to-understand manner. Further, when the EMERGENCY information is acquired, the other device icon is displayed accordingly, so that a user carrying another mobile terminal device can be notified that it is in an emergency state. Further, when the EMERGENCY MAN-DOWN information is acquired, the other device icon is displayed accordingly, so that it is possible to notify that the user carrying the other mobile terminal device is in the emergency man down state.
Moreover, since the image which imaged the user who carried the other mobile terminal device which transmitted EMERGENCY MAN-DOWN information is transmitted from another mobile terminal device, since the said user has fallen, it is hidden in the obstacle Even the presence of the user can be confirmed. In addition, the other device icon is displayed by converting the position so that it can be displayed, and the relationship between the positions before and after the conversion is also displayed, so even if the other device icon is hidden by an obstacle, the other device icon is displayed. The correct position can be notified while displaying. Also, the state, location, distance, etc. of each mobile terminal device including emergency can be confirmed in augmented reality. In addition, the emergency state can be displayed intuitively and easily in a state where the mobile terminal device is surrounded.
Next, Example 3 will be described. The mobile terminal device according to the third embodiment displays the other device icon indicating the other mobile terminal device on the captured image by using the AR technology as before. On the other hand, the mobile terminal device according to the third embodiment displays the movement trajectory of the other device icon and the movement trajectory of the icon indicating the mobile terminal device (hereinafter referred to as “own device icon”) on the captured image. . At that time, signal strength information is also shown. Thus, by indicating the movement trajectory and the signal strength, a determination material for a position where communication quality is likely to be improved is provided. The communication system 100 and the mobile terminal apparatus 10 according to the third embodiment are of the same type as those shown in FIGS. Here, it demonstrates centering on the difference from before.
The storage unit 30 in FIG. 2 stores the history of the first position information and the history of the second position information. The storage unit 30 also stores a signal intensity corresponding to the history of the second position information. This is the signal intensity measured by the acquisition unit 24 when the communication unit 22 receives a signal including the second position information. FIG. 12 shows a data structure of a database stored in the storage unit 30 according to the third embodiment of the present invention. As shown in the figure, a history of information of “name”, “group”, “state”, “address”, “direction”, “update”, “signal strength”, and “time” is stored for each mobile terminal device 10. The Since the processing in the first mobile terminal apparatus 10a is an object of description, “signal strength” is not indicated in the history of the first mobile terminal apparatus 10a. Returning to FIG.
In the image captured by the image capturing unit 26, the processing unit 28 displays the other device icon in each of the history of the first position information stored in the storage unit 30 and the coordinate history derived based on the history of the second position information. 200 is arranged. This process corresponds to executing the same process until now for each of a plurality of coordinates. Further, the processing unit 28 arranges the own device icon indicating the mobile terminal device 10 in each of the coordinate history derived based on the history of the first position information stored in the storage unit 30. This process corresponds to executing the process so far only on the first position information.
Further, the processing unit 28 identifies the other device icon 200 and the own device icon that can correspond to each other among the plurality of other device icons 200 and the plurality of own device icons, for example, those whose time is included in a predetermined range. . The processing unit 28 draws a display (hereinafter referred to as a “past correspondence line”) for associating the other device icon 200 and the own device icon between the identified other device icon 200 and the own device icon. An image 50 is generated.
FIG. 13 shows an image 50 generated by the processing unit 28 according to the third embodiment of the present invention. As illustrated, according to the history, a plurality of second other device icons 200b are shown, and a plurality of own device icons 220 are shown. As described above, the second other device icon 200b corresponds to the second mobile terminal device 10b, and the own device icon 220 corresponds to the first mobile terminal device 10a. Also, one second other device icon 200b and one device icon 220 are connected by a past correspondence line 222. Here, for the sake of clarity, only one past correspondence line 222 is shown, but a plurality of past correspondence lines 222 may be shown. Note that the display mode of the past correspondence line 222 may be changed according to the signal strength, similarly to the transmission line 204.
By such display, it is possible to confirm the mutual relationship between the other mobile terminal device 10 and the mobile terminal device 10. If the other device icon 200 or the own device icon 220 overlaps the trajectory, a new one is placed in front. When the other device icon 200 and the own device icon 220 overlap, it is possible to instruct which one is displayed in front by setting. Furthermore, by tapping the other device icon 200 or the own device icon 220, information on the communication system 100 at that time and signal strength at that time may be displayed. By displaying in this way, it is possible to easily find a position where the communication quality is improved with reference to a position where good communication was possible in the past when the signal strength at the current position is weak. Further, in the patrol monitoring business and the like, since the user himself / herself of the mobile terminal device 10 and the past route of the communication partner can be easily confirmed, it can be used for the formulation or change of the future tour route. For example, when patrol monitoring or the like is performed using a plurality of mobile terminal apparatuses 10, it is possible to perform efficient patrols while avoiding routes that the other mobile terminal apparatuses 10 have visited within the past predetermined time.
According to this embodiment, since the history of the position coordinates of the mobile terminal device and the position coordinates of the other mobile terminal devices are stored, the movement locus of the other device icon and the movement locus of the own device icon are displayed on the image. Can be displayed. In addition, a past correspondence line that associates the other device icon and the own device icon that can correspond to each other among the plurality of other device icons and the plurality of own device icons is displayed, so that it is possible to know at which timing and at which position the communication was performed. Easy to display. In addition, since the signal strength is also displayed, it is possible to display in an easy-to-understand manner at which timing, at which position, and with what communication quality. In addition, since it is displayed in an easy-to-understand manner at what timing, at what position, and with what communication quality, when the communication quality deteriorates, it is easy to return to a state where the communication quality is good.
Next, Example 4 will be described. The mobile terminal device according to the fourth embodiment displays the other device icon indicating the other mobile terminal device on the captured image by using the AR technology as before. The fourth embodiment relates to outputting an instruction to another mobile terminal device based on information shown on an image. The communication system 100 and the mobile terminal apparatus 10 according to the fourth embodiment are of the same type as those shown in FIGS. Here, it demonstrates centering on the difference with Example 1. FIG.
FIG. 14 shows an image 50 generated by the processing unit 28 according to the fourth embodiment of the present invention. In the image 50, the second other device icon 200b and the second transmission line 204b are shown as in FIG. When the user taps the vicinity of the second other device icon 200b in the image 50 displayed on the display unit 32, the submenu 230 is displayed. Further, the user selects “movement instruction” in the submenu 230. Subsequently, when the user taps a position where the user of the second mobile terminal apparatus 10b is to be moved, the destination icon 232 is displayed on the image 50. The processing unit 28 uses the first position information stored in the storage unit 30 to derive latitude / longitude (third position information) based on the coordinates of the destination icon 232, and derives the derived latitude / longitude (first position information). 3 position information) or movement instruction information based on the third position information is transmitted to the second mobile terminal apparatus 10b via the communication unit 22.
When the second mobile terminal apparatus 10b receives the movement instruction, the second mobile terminal apparatus 10b displays a navigation screen showing a movement route to the destination on the display unit 32. Further, the processing unit 28 of the first mobile terminal apparatus 10a calculates a relative position with respect to the destination (third position information) based on the current position (second position information) of the second mobile terminal apparatus 10b, and expands it. A text message such as “Please move 30 m to the east” or an automatic voice message may be generated without using the real screen or the map screen. These messages are also transmitted to the second mobile terminal apparatus 10b.
According to the present embodiment, since the position to be moved on the image is indicated, a movement instruction can be easily output. In addition, when a movement instruction is received, a navigation screen up to the position indicated by the movement instruction is displayed, so that the movement can be easily performed. In addition, since the signal strength is shown on the image, the other mobile terminal apparatus 10 can be guided to a position where the communication quality is likely to be improved.
According to the first to fourth embodiments, the communication system 100 uses a business wireless system. However, the present invention is not limited to this. For example, a wireless communication system other than a commercial wireless system may be used. According to this modification, the degree of freedom of the system can be improved.
DESCRIPTION OF SYMBOLS 10 Mobile terminal device, 20 Positioning part, 22 Communication part, 24 Acquisition part, 26 Imaging part, 28 Processing part, 30 Storage part, 32 Display part, 100 Communication system.
A mobile terminal device,
A positioning unit for positioning the first position information of the mobile terminal device;
A communication unit that acquires second position information measured in the other mobile terminal device by communicating with the other mobile terminal device;
An imaging unit for imaging the environment around the mobile terminal device;
An acquisition unit for acquiring information on a state depending on an environment around the other mobile terminal device;
On the image captured by the image capturing unit, the other mobile terminal device indicates coordinates derived based on the first position information measured by the positioning unit and the second position information acquired by the communication unit. A display unit that displays the other device icon to be displayed, and displays the information acquired in the acquisition unit in association with the other device icon;
The acquisition unit measures signal strength when the communication unit receives a signal from the other mobile terminal device as information on a state depending on an environment around the other mobile terminal device,
The mobile terminal apparatus according to claim 1, wherein the display unit displays the signal intensity measured in the acquisition unit in association with the other apparatus icon.
The acquisition unit acquires the signal strength measured in the other mobile terminal device as information on a state depending on the environment around the other mobile terminal device,
The mobile terminal device according to claim 1, wherein the display unit displays the signal intensity acquired by the acquisition unit in association with the other device icon.
The mobile terminal apparatus according to claim 2 or 3, wherein the display unit changes a display mode of the signal strength according to a value of the signal strength measured in the acquisition unit.
The acquisition unit acquires state information transmitted from the other mobile terminal device and received by the communication unit as information related to a state depending on an environment around the other mobile terminal device;
The mobile terminal device according to claim 1, wherein the display unit displays the state information acquired in the acquisition unit in association with the other device icon.
The display unit displays the other device icon at another coordinate different from the coordinates derived based on the first position information and the second position information, and relates to a relationship between the another coordinate and the coordinate. Information is also displayed, The mobile terminal device of Claim 5 characterized by the above-mentioned.
In addition to storing the history of the first position information, the storage unit further stores a history of the second position information,
In the image captured by the image capturing unit, the display unit includes a coordinate history derived based on the history of the first position information and the history of the second position information stored in the storage unit. In addition to displaying a device icon, each of the coordinates history derived based on the history of the stored first position information, the device icon indicating the mobile terminal device is displayed,
The said display part performs the display for associating the other apparatus icon and self apparatus icon which can respond | correspond mutually among a plurality of other apparatus icons and a plurality of own apparatus icons. Mobile terminal device.
An input unit for allowing a user to specify coordinates on an image displayed on the display unit;
A processing unit for deriving third position information corresponding to the coordinates based on the coordinates specified by the input unit and the first position information measured by the positioning unit;
The mobile terminal apparatus according to claim 1, wherein the communication unit transmits information related to third position information to the other mobile terminal apparatus.
A program for a mobile terminal device,
Positioning the first location information of the mobile terminal device;
Obtaining second position information measured in the other mobile terminal device by communicating with the other mobile terminal device;
Imaging the environment around the mobile terminal device;
Obtaining information on a state dependent on the environment around the other mobile terminal device;
In the captured image, the other device icon indicating the other mobile terminal device is displayed at the coordinates derived based on the first position information obtained by positioning and the obtained second position information, and the obtained information is displayed. Displaying in association with the other device icon;
JP2016023407A 2016-02-10 2016-02-10 Mobile terminal device and program Pending JP2017143409A (en)
JP2016023407A JP2017143409A (en) 2016-02-10 2016-02-10 Mobile terminal device and program
US15/427,566 US10176618B2 (en) 2016-02-10 2017-02-08 Mobile terminal device, method, and storage medium for displaying captured images
JP2017143409A true JP2017143409A (en) 2017-08-17
ID=59497864
JP2016023407A Pending JP2017143409A (en) 2016-02-10 2016-02-10 Mobile terminal device and program
US (1) US10176618B2 (en)
JP (1) JP2017143409A (en)
JP2012221330A (en) 2011-04-12 2012-11-12 Shigemitsu Takaishi Matching system
2016-02-10 JP JP2016023407A patent/JP2017143409A/en active Pending
2017-02-08 US US15/427,566 patent/US10176618B2/en active Active
US20170228912A1 (en) 2017-08-10
US10176618B2 (en) 2019-01-08
EP2520104A1 (en) 2012-11-07 Directional animation for communications
US8352178B2 (en) 2013-01-08 Method and device for generating and communicating geographic route information between wireless communication devices
CN103105993B (en) 2015-05-20 Method and system for realizing interaction based on augmented reality technology
2019-11-12 A977 Report on retrieval
2019-11-19 A131 Notification of reasons for refusal