Source: https://patents.google.com/patent/US9608725B2/en
Timestamp: 2019-08-23 16:56:26
Document Index: 700047469

Matched Legal Cases: ['Application No. 62', 'Application No. 62', 'Application No. 2014', 'Application No. 2014', 'Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 2013', 'Application No. 2013', 'Application No. 2013', 'Application No. 2013', 'Application No. 2013', 'Application No. 2013', 'Application No. 2013', 'Application No. 2013', 'Application No. 2013', 'Application No. 2012', 'art 7513', 'art 7514', 'art 7515', 'art 7515', 'art 7515']

US9608725B2 - Information processing program, reception program, and information processing apparatus - Google Patents
Information processing program, reception program, and information processing apparatus Download PDF
US9608725B2
US9608725B2 US14/582,751 US201414582751A US9608725B2 US 9608725 B2 US9608725 B2 US 9608725B2 US 201414582751 A US201414582751 A US 201414582751A US 9608725 B2 US9608725 B2 US 9608725B2
US14/582,751
US20150188632A1 (en
Akihiro Ueki
Kengo MIYOSHI
2012-12-27 Priority to US201261746315P priority Critical
2012-12-27 Priority to JP2012286339 priority
2012-12-27 Priority to JP2012-286339 priority
2013-03-28 Priority to JP2013070740 priority
2013-03-28 Priority to JP2013-070740 priority
2013-03-28 Priority to US201361805978P priority
2013-04-10 Priority to US201361810291P priority
2013-04-10 Priority to JP2013-082546 priority
2013-04-10 Priority to JP2013082546 priority
2013-05-24 Priority to JP2013110445 priority
2013-05-24 Priority to JP2013-110445 priority
2013-07-30 Priority to JP2013158359 priority
2013-07-30 Priority to JP2013-158359 priority
2013-07-30 Priority to US201361859902P priority
2013-08-30 Priority to JP2013-180729 priority
2013-08-30 Priority to JP2013180729 priority
2013-08-30 Priority to US201361872028P priority
2013-10-25 Priority to JP2013-222827 priority
2013-10-25 Priority to JP2013222827 priority
2013-10-25 Priority to US201361895615P priority
2013-10-29 Priority to JP2013-224805 priority
2013-10-29 Priority to US201361896879P priority
2013-10-29 Priority to JP2013224805 priority
2013-11-15 Priority to US201361904611P priority
2013-11-15 Priority to JP2013-237460 priority
2013-11-15 Priority to JP2013237460 priority
2013-11-22 Priority to JP2013-242407 priority
2013-11-22 Priority to JP2013242407 priority
2013-12-27 Priority to US14/142,413 priority patent/US9341014B2/en
2014-07-01 Priority to US201462019515P priority
2014-07-25 Priority to US201462028991P priority
2014-09-19 Priority to JP2014-192032 priority
2014-09-19 Priority to JP2014192032 priority
2014-11-14 Priority to JP2014232187 priority
2014-11-14 Priority to JP2014-232187 priority
2014-12-24 Priority to US14/582,751 priority patent/US9608725B2/en
2014-12-24 Application filed by Panasonic Intellectual Property Corp filed Critical Panasonic Intellectual Property Corp
2015-02-17 Assigned to PANASONIC INTELLECTUAL PROPERTY CORPORATION OF AMERICA reassignment PANASONIC INTELLECTUAL PROPERTY CORPORATION OF AMERICA ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: OSHIMA, MITSUAKI, NAKANISHI, KOJI, MIYOSHI, KENGO, MUKAI, TSUTOMU, AOYAMA, HIDEKI, MAEDA, TOSHIYUKI, UEKI, AKIHIRO
2015-07-02 Publication of US20150188632A1 publication Critical patent/US20150188632A1/en
2015-12-18 Priority claimed from US14/973,783 external-priority patent/US9608727B2/en
2016-12-16 Priority claimed from US15/381,940 external-priority patent/US10303945B2/en
2017-03-28 Publication of US9608725B2 publication Critical patent/US9608725B2/en
2017-11-15 Priority claimed from US15/813,244 external-priority patent/US10361780B2/en
2017-12-15 Priority claimed from US15/843,790 external-priority patent/US20180212684A1/en
An information processing program causes a computer to execute: encoding information to be transmitted, to determine a luminance change frequency; and outputting a signal of the determined luminance change frequency to cause a light emitter to change in luminance according to the determined luminance change frequency to transmit the information. In the encoding, each of a first frequency and a second frequency different from the first frequency is determined as the luminance change frequency. In the outputting, each of a signal of the first frequency and a signal of the second frequency is output as the signal of the determined luminance change frequency, to cause the light emitter to change in luminance according to the first frequency during a first time and change in luminance according to the second frequency during a second time different from the first time after the first time elapses.
The present application claims the benefit of U.S. Provisional Patent Application No. 62/019,515 filed on Jul. 1, 2014, U.S. Provisional Patent Application No. 62/028,991 filed on Jul. 25, 2014, Japanese Patent Application No. 2014-192032 filed on Sep. 19, 2014, and Japanese Patent Application No. 2014-232187 filed on Nov. 14, 2014, is a continuation-in-part of U.S. Non-Provisional patent application Ser. No. 14/142,413 filed on Dec. 27, 2013, which claims benefit of U.S. Provisional Patent Application No. 61/904,611 filed on Nov. 15, 2013, U.S. Provisional Patent Application No. 61/896,879 filed on Oct. 29, 2013, U.S. Provisional Patent Application No. 61/895,615 filed on Oct. 25, 2013, U.S. Provisional Patent Application No. 61/872,028 filed on Aug. 30, 2013, U.S. Provisional Patent Application No. 61/859,902 filed on Jul. 30, 2013, U.S. Provisional Patent Application No. 61/810,291 filed on Apr. 10, 2013, U.S. Provisional Patent Application No. 61/805,978 filed on Mar. 28, 2013, U.S. Provisional Patent Application No. 61/746,315 filed on Dec. 27, 2012, Japanese Patent Application No. 2013-242407 filed on Nov. 22, 2013, Japanese Patent Application No. 2013-237460 filed on Nov. 15, 2013, Japanese Patent Application No. 2013-224805 filed on Oct. 29, 2013, Japanese Patent Application No. 2013-222827 filed on Oct. 25, 2013, Japanese Patent Application No. 2013-180729 filed on Aug. 30, 2013, Japanese Patent Application No. 2013-158359 filed on Jul. 30, 2013, Japanese Patent Application No. 2013-110445 filed on May 24, 2013, Japanese Patent Application No. 2013-082546 filed on Apr. 10, 2013, Japanese Patent Application No. 2013-070740 filed on Mar. 28, 2013, and Japanese Patent Application No. 2012-286339 filed on Dec. 27, 2012. The entire disclosures of the above-identified applications, including the specifications, drawings and claims are incorporated herein by reference in their entirety.
The present disclosure relates to a method of communication between a mobile terminal such as a smartphone, a tablet terminal, or a mobile phone and a home electric appliance such as an air conditioner, a lighting device, or a rice cooker.
In recent years, a home-electric-appliance cooperation function has been introduced for a home network, with which various home electric appliances are connected to a network by a home energy management system (HEMS) having a function of managing power usage for addressing an environmental issue, turning power on/off from outside a house, and the like, in addition to cooperation of AV home electric appliances by internet protocol (IP) connection using Ethernet® or wireless local area network (LAN). However, there are home electric appliances whose computational performance is insufficient to have a communication function, and home electric appliances which do not have a communication function due to a matter of cost.
In order to solve such a problem, Patent Literature (PTL) 1 discloses a technique of efficiently establishing communication between devices among limited optical spatial transmission devices which transmit information to a free space using light, by performing communication using plural single color light sources of illumination light.
[Patent Literature 1] Japanese Unexamined Patent Application Publication No. 2002-290335
However, the conventional method is limited to a case in which a device to which the method is applied has three color light sources such as an illuminator. One non-limiting and exemplary embodiment provides an information processing program or a reception program that solves this problem and enables communication between various devices including a device with low computational performance.
An information processing program according to an aspect of the present disclosure is an information processing program for causing a computer to process information to be transmitted, in order for the information to be transmitted by way of luminance change, and causes the computer to execute: encoding the information to determine a luminance change frequency; and outputting a signal of the luminance change frequency determined, to cause a light emitter to change in luminance according to the luminance change frequency determined, to transmit the information, wherein in the encoding, each of a first frequency and a second frequency different from the first frequency is determined as the luminance change frequency, and in the outputting, each of a signal of the first frequency and a signal of the second frequency is output as the signal of the luminance change frequency determined, to cause the light emitter to change in luminance according to the first frequency during a first time and change in luminance according to the second frequency during a second time after the first time elapses, the second time being different from the first time.
An information communication method and a reception program disclosed herein provide an information processing program and a reception program that enable communication between various devices including a device with low computational performance.
FIG. 1 is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1.
FIG. 2 is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1.
FIG. 3 is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1.
FIG. 4 is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1.
FIG. 5A is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1.
FIG. 5B is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1.
FIG. 5C is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1.
FIG. 5D is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1.
FIG. 5E is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1.
FIG. 5F is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1.
FIG. 5G is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1.
FIG. 5H is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1.
FIG. 6A is a flowchart of an information communication method in Embodiment 1.
FIG. 6B is a block diagram of an information communication device in Embodiment 1.
FIG. 7 is a diagram illustrating an example of each mode of a receiver in Embodiment 2.
FIG. 8 is a diagram illustrating an example of imaging operation of a receiver in Embodiment 2.
FIG. 9 is a diagram illustrating another example of imaging operation of a receiver in Embodiment 2.
FIG. 10A is a diagram illustrating another example of imaging operation of a receiver in Embodiment 2.
FIG. 10B is a diagram illustrating another example of imaging operation of a receiver in Embodiment 2.
FIG. 10C is a diagram illustrating another example of imaging operation of a receiver in Embodiment 2.
FIG. 11A is a diagram illustrating an example of camera arrangement of a receiver in Embodiment 2.
FIG. 11B is a diagram illustrating another example of camera arrangement of a receiver in Embodiment 2.
FIG. 12 is a diagram illustrating an example of display operation of a receiver in Embodiment 2.
FIG. 13 is a diagram illustrating an example of display operation of a receiver in Embodiment 2.
FIG. 14 is a diagram illustrating an example of operation of a receiver in Embodiment 2.
FIG. 15 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 16 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 17 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 18 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 19 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 20 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 21 is a diagram illustrating an example of operation of a receiver, a transmitter, and a server in Embodiment 2.
FIG. 22 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 23 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 24 is a diagram illustrating an example of initial setting of a receiver in Embodiment 2.
FIG. 25 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 26 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 27 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 28 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 29 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 30 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 31A is a diagram illustrating a pen used to operate a receiver in Embodiment 2.
FIG. 31B is a diagram illustrating operation of a receiver using a pen in Embodiment 2.
FIG. 32 is a diagram illustrating an example of appearance of a receiver in Embodiment 2.
FIG. 33 is a diagram illustrating another example of appearance of a receiver in Embodiment 2.
FIG. 34 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 35A is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 35B is a diagram illustrating an example of application using a receiver in Embodiment 2.
FIG. 36A is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 36B is a diagram illustrating an example of application using a receiver in Embodiment 2.
FIG. 37A is a diagram illustrating an example of operation of a transmitter in Embodiment 2.
FIG. 37B is a diagram illustrating another example of operation of a transmitter in Embodiment 2.
FIG. 38 is a diagram illustrating another example of operation of a transmitter in Embodiment 2.
FIG. 39 is a diagram illustrating another example of operation of a transmitter in Embodiment 2.
FIG. 40 is a diagram illustrating an example of communication form between a plurality of transmitters and a receiver in Embodiment 2.
FIG. 41 is a diagram illustrating an example of operation of a plurality of transmitters in Embodiment 2.
FIG. 42 is a diagram illustrating another example of communication form between a plurality of transmitters and a receiver in Embodiment 2.
FIG. 43 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 44 is a diagram illustrating an example of application of a receiver in Embodiment 2.
FIG. 45 is a diagram illustrating an example of application of a receiver in Embodiment 2.
FIG. 46 is a diagram illustrating an example of application of a receiver in Embodiment 2.
FIG. 47 is a diagram illustrating an example of application of a transmitter in Embodiment 2.
FIG. 48 is a diagram illustrating an example of application of a transmitter in Embodiment 2.
FIG. 49 is a diagram illustrating an example of application of a reception method in Embodiment 2.
FIG. 50 is a diagram illustrating an example of application of a transmitter in Embodiment 2.
FIG. 51 is a diagram illustrating an example of application of a transmitter in Embodiment 2.
FIG. 52 is a diagram illustrating an example of application of a transmitter in Embodiment 2.
FIG. 53 is a diagram illustrating another example of operation of a receiver in Embodiment 2.
FIG. 54 is a flowchart illustrating an example of operation of a receiver in Embodiment 3.
FIG. 55 is a flowchart illustrating another example of operation of a receiver in Embodiment 3.
FIG. 56A is a block diagram illustrating an example of a transmitter in Embodiment 3.
FIG. 56B is a block diagram illustrating another example of a transmitter in Embodiment 3.
FIG. 57 is a diagram illustrating an example of a structure of a system including a plurality of transmitters in Embodiment 3.
FIG. 58 is a block diagram illustrating another example of a transmitter in Embodiment 3.
FIG. 59A is a diagram illustrating an example of a transmitter in Embodiment 3.
FIG. 59B is a diagram illustrating an example of a transmitter in Embodiment 3.
FIG. 59C is a diagram illustrating an example of a transmitter in Embodiment 3.
FIG. 60A is a diagram illustrating an example of a transmitter in Embodiment 3.
FIG. 60B is a diagram illustrating an example of a transmitter in Embodiment 3.
FIG. 61 is a diagram illustrating an example of processing operation of a receiver, a transmitter, and a server in Embodiment 3.
FIG. 62 is a diagram illustrating an example of processing operation of a receiver, a transmitter, and a server in Embodiment 3.
FIG. 63 is a diagram illustrating an example of processing operation of a receiver, a transmitter, and a server in Embodiment 3.
FIG. 64A is a diagram for describing synchronization between a plurality of transmitters in Embodiment 3.
FIG. 64B is a diagram for describing synchronization between a plurality of transmitters in Embodiment 3.
FIG. 65 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 3.
FIG. 66 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 3.
FIG. 67 is a diagram illustrating an example of operation of a transmitter, a receiver, and a server in Embodiment 3.
FIG. 68 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 3.
FIG. 69 is a diagram illustrating an example of appearance of a receiver in Embodiment 3.
FIG. 70 is a diagram illustrating an example of operation of a transmitter, a receiver, and a server in Embodiment 3.
FIG. 71 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 3.
FIG. 72 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 3.
FIG. 73 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 3.
FIG. 74 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 3.
FIG. 75A is a diagram illustrating another example of a structure of information transmitted by a transmitter in Embodiment 3.
FIG. 75B is a diagram illustrating another example of a structure of information transmitted by a transmitter in Embodiment 3.
FIG. 76 is a diagram illustrating an example of a 4-value PPM modulation scheme by a transmitter in Embodiment 3.
FIG. 77 is a diagram illustrating an example of a PPM modulation scheme by a transmitter in Embodiment 3.
FIG. 78 is a diagram illustrating an example of a PPM modulation scheme by a transmitter in Embodiment 3.
FIG. 79A is a diagram illustrating an example of a luminance change pattern corresponding to a header (preamble unit) in Embodiment 3.
FIG. 79B is a diagram illustrating an example of a luminance change pattern in Embodiment 3.
FIG. 80A is a diagram illustrating an example of a luminance change pattern in Embodiment 3.
FIG. 80B is a diagram illustrating an example of a luminance change pattern in Embodiment 3.
FIG. 81 is a diagram illustrating an example of operation of a receiver in an in-front-of-store situation in Embodiment 4.
FIG. 82 is a diagram illustrating anther example of operation of a receiver in an in-front-of-store situation in Embodiment 4.
FIG. 83 is a diagram illustrating an example of next operation of a receiver in an in-front-of-store situation in Embodiment 4.
FIG. 84 is a diagram illustrating an example of next operation of a receiver in an in-front-of-store situation in Embodiment 4.
FIG. 85 is a diagram illustrating an example of next operation of a receiver in an in-front-of-store situation in Embodiment 4.
FIG. 86 is a diagram illustrating an example of operation of a display device in an in-front-of-store situation in Embodiment 4.
FIG. 87 is a diagram illustrating an example of next operation of a display device in an in-front-of-store situation in Embodiment 4.
FIG. 88 is a diagram illustrating an example of next operation of a display device in an in-front-of-store situation in Embodiment 4.
FIG. 89 is a diagram illustrating an example of next operation of a receiver in an in-front-of-store situation in Embodiment 4.
FIG. 90 is a diagram illustrating an example of next operation of a receiver in an in-front-of-store situation in Embodiment 4.
FIG. 91 is a diagram illustrating an example of next operation of a receiver in an in-front-of-store situation in Embodiment 4.
FIG. 92 is a diagram illustrating an example of next operation of a receiver in an in-front-of-store situation in Embodiment 4.
FIG. 93 is a diagram illustrating an example of next operation of a receiver in an in-front-of-store situation in Embodiment 4.
FIG. 94 is a diagram illustrating an example of next operation of a receiver in an in-front-of-store situation in Embodiment 4.
FIG. 95 is a diagram illustrating an example of operation of a receiver in a store search situation in Embodiment 4.
FIG. 96 is a diagram illustrating an example of next operation of a receiver in a store search situation in Embodiment 4.
FIG. 97 is a diagram illustrating an example of next operation of a receiver in a store search situation in Embodiment 4.
FIG. 98 is a diagram illustrating an example of operation of a receiver in a movie advertisement situation in Embodiment 4.
FIG. 99 is a diagram illustrating an example of next operation of a receiver in a movie advertisement situation in Embodiment 4.
FIG. 100 is a diagram illustrating an example of next operation of a receiver in a movie advertisement situation in Embodiment 4.
FIG. 101 is a diagram illustrating an example of next operation of a receiver in a movie advertisement situation in Embodiment 4.
FIG. 102 is a diagram illustrating an example of operation of a receiver in a museum situation in Embodiment 4.
FIG. 103 is a diagram illustrating an example of next operation of a receiver in a museum situation in Embodiment 4.
FIG. 104 is a diagram illustrating an example of next operation of a receiver in a museum situation in Embodiment 4.
FIG. 105 is a diagram illustrating an example of next operation of a receiver in a museum situation in Embodiment 4.
FIG. 106 is a diagram illustrating an example of next operation of a receiver in a museum situation in Embodiment 4.
FIG. 107 is a diagram illustrating an example of next operation of a receiver in a museum situation in Embodiment 4.
FIG. 108 is a diagram illustrating an example of operation of a receiver in a bus stop situation in Embodiment 4.
FIG. 109 is a diagram illustrating an example of next operation of a receiver in a bus stop situation in Embodiment 4.
FIG. 110 is a diagram for describing imaging in Embodiment 4.
FIG. 111 is a diagram for describing transmission and imaging in Embodiment 4.
FIG. 112 is a diagram for describing transmission in Embodiment 4.
FIG. 113 is a diagram illustrating an example of operation of a transmitter in Embodiment 5.
FIG. 114 is a diagram illustrating an example of operation of a transmitter in Embodiment 5.
FIG. 115 is a diagram illustrating an example of operation of a transmitter in Embodiment 5.
FIG. 116 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 117 is a diagram illustrating an example of operation of a receiver in Embodiment 5.
FIG. 118 is a diagram illustrating an example of operation of a receiver in Embodiment 5.
FIG. 119 is a diagram illustrating an example of operation of a system including a transmitter, a receiver, and a server in Embodiment 5.
FIG. 120 is a block diagram illustrating a structure of a transmitter in Embodiment 5.
FIG. 121 is a block diagram illustrating a structure of a receiver in Embodiment 5.
FIG. 122 is a diagram illustrating an example of operation of a transmitter in Embodiment 5.
FIG. 123 is a diagram illustrating an example of operation of a transmitter in Embodiment 5.
FIG. 124 is a diagram illustrating an example of operation of a transmitter in Embodiment 5.
FIG. 125 is a diagram illustrating an example of operation of a transmitter in Embodiment 5.
FIG. 126 is a diagram illustrating an example of operation of a transmitter in Embodiment 5.
FIG. 127 is a diagram illustrating an example of operation of a transmitter in Embodiment 5.
FIG. 128 is a diagram illustrating an example of operation of a transmitter in Embodiment 5.
FIG. 129 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 130 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 131 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 132 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 133 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 134 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 135 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 136 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 137 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 138 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 139 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 140 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 141 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 142 is a diagram illustrating a coding scheme in Embodiment 5.
FIG. 143 is a diagram illustrating a coding scheme that can receive light even in the case of capturing an image in an oblique direction in Embodiment 5.
FIG. 144 is a diagram illustrating a coding scheme that differs in information amount depending on distance in Embodiment 5.
FIG. 145 is a diagram illustrating a coding scheme that differs in information amount depending on distance in Embodiment 5.
FIG. 146 is a diagram illustrating a coding scheme that divides data in Embodiment 5.
FIG. 147 is a diagram illustrating an opposite-phase image insertion effect in Embodiment 5.
FIG. 148 is a diagram illustrating an opposite-phase image insertion effect in Embodiment 5.
FIG. 149 is a diagram illustrating a superresolution process in Embodiment 5.
FIG. 150 is a diagram illustrating a display indicating visible light communication capability in Embodiment 5.
FIG. 151 is a diagram illustrating information obtainment using a visible light communication signal in Embodiment 5.
FIG. 152 is a diagram illustrating a data format in Embodiment 5.
FIG. 153 is a diagram illustrating reception by estimating a stereoscopic shape in Embodiment 5.
FIG. 154 is a diagram illustrating reception by estimating a stereoscopic shape in Embodiment 5.
FIG. 155 is a diagram illustrating stereoscopic projection in Embodiment 5.
FIG. 156 is a diagram illustrating stereoscopic projection in Embodiment 5.
FIG. 157 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 158 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 5.
FIG. 159 is a diagram illustrating an example of a transmission signal in Embodiment 6.
FIG. 160 is a diagram illustrating an example of a transmission signal in Embodiment 6.
FIG. 161A is a diagram illustrating an example of an image (bright line image) captured by a receiver in Embodiment 6.
FIG. 161B is a diagram illustrating an example of an image (bright line image) captured by a receiver in Embodiment 6.
FIG. 161C is a diagram illustrating an example of an image (bright line image) captured by a receiver in Embodiment 6.
FIG. 162A is a diagram illustrating an example of an image (bright line image) captured by a receiver in Embodiment 6.
FIG. 162B is a diagram illustrating an example of an image (bright line image) captured by a receiver in Embodiment 6.
FIG. 163A is a diagram illustrating an example of an image (bright line image) captured by a receiver in Embodiment 6.
FIG. 163B is a diagram illustrating an example of an image (bright line image) captured by a receiver in Embodiment 6.
FIG. 163C is a diagram illustrating an example of an image (bright line image) captured by a receiver in Embodiment 6.
FIG. 164 is a diagram illustrating an example of an image (bright line image) captured by a receiver in Embodiment 6.
FIG. 165 is a diagram illustrating an example of a transmission signal in Embodiment 6.
FIG. 166 is a diagram illustrating an example of operation of a receiver in Embodiment 6.
FIG. 167 is a diagram illustrating an example of an instruction to a user displayed on a screen of a receiver in Embodiment 6.
FIG. 168 is a diagram illustrating an example of an instruction to a user displayed on a screen of a receiver in Embodiment 6.
FIG. 169 is a diagram illustrating an example of a signal transmission method in Embodiment 6.
FIG. 170 is a diagram illustrating an example of a signal transmission method in Embodiment 6.
FIG. 171 is a diagram illustrating an example of a signal transmission method in Embodiment 6.
FIG. 172 is a diagram illustrating an example of a signal transmission method in Embodiment 6.
FIG. 173 is a diagram for describing a use case in Embodiment 6.
FIG. 174 is a diagram illustrating an information table transmitted from a smartphone to a server in Embodiment 6.
FIG. 175 is a block diagram of a server in Embodiment 6.
FIG. 176 is a flowchart illustrating an overall process of a system in Embodiment 6.
FIG. 177 is a diagram illustrating an information table transmitted from a server to a smartphone in Embodiment 6.
FIG. 178 is a diagram illustrating flow of screen displayed on a wearable device from when a user receives information from a server in front of a store to when the user actually buys a product in Embodiment 6.
FIG. 179 is a diagram for describing another use case in Embodiment 6.
FIG. 180 is a diagram illustrating a service provision system using the reception method described in any of the foregoing embodiments.
FIG. 181 is a flowchart illustrating service provision flow.
FIG. 182 is a flowchart illustrating service provision in another example.
FIG. 183 is a flowchart illustrating service provision in another example.
FIG. 184A is a diagram for describing a modulation scheme that facilitates reception in Embodiment 8.
FIG. 184B is a diagram for describing a modulation scheme that facilitates reception in Embodiment 8.
FIG. 185 is a diagram for describing a modulation scheme that facilitates reception in Embodiment 8.
FIG. 186 is a diagram for describing communication using bright lines and image recognition in Embodiment 8.
FIG. 187A is a diagram for describing an imaging element use method suitable for visible light signal reception in Embodiment 8.
FIG. 187B is a diagram for describing an imaging element use method suitable for visible light signal reception in Embodiment 8.
FIG. 187C is a diagram for describing an imaging element use method suitable for visible light signal reception in Embodiment 8.
FIG. 187D is a diagram for describing an imaging element use method suitable for visible light signal reception in Embodiment 8.
FIG. 187E is a flowchart for describing an imaging element use method suitable for visible light signal reception in Embodiment 8.
FIG. 188 is a diagram illustrating a captured image size suitable for visible light signal reception in Embodiment 8.
FIG. 189A is a diagram illustrating a captured image size suitable for visible light signal reception in Embodiment 8.
FIG. 189B is a flowchart illustrating operation for switching to a captured image size suitable for visible light signal reception in Embodiment 8.
FIG. 189C is a flowchart illustrating operation for switching to a captured image size suitable for visible light signal reception in Embodiment 8.
FIG. 190 is a diagram for describing visible light signal reception using zoom in Embodiment 8.
FIG. 191 is a diagram for describing an image data size reduction method suitable for visible light signal reception in Embodiment 8.
FIG. 192 is a diagram for describing a modulation scheme with high reception error detection accuracy in Embodiment 8.
FIG. 193 is a diagram for describing a change of operation of a receiver according to situation in Embodiment 8.
FIG. 194 is a diagram for describing notification of visible light communication to humans in Embodiment 8.
FIG. 195 is a diagram for describing expansion in reception range by a diffusion plate in Embodiment 8.
FIG. 196 is a diagram for describing a method of synchronizing signal transmission from a plurality of projectors in Embodiment 8.
FIG. 197 is a diagram for describing a method of synchronizing signal transmission from a plurality of displays in Embodiment 8.
FIG. 198 is a diagram for describing visible light signal reception by an illuminance sensor and an image sensor in Embodiment 8.
FIG. 199 is a diagram for describing a reception start trigger in Embodiment 8.
FIG. 200 is a diagram for describing a reception start gesture in Embodiment 8.
FIG. 201 is a diagram for describing an example of application to a car navigation system in Embodiment 8.
FIG. 202 is a diagram for describing an example of application to a car navigation system in Embodiment 8.
FIG. 203 is a diagram for describing an example of application to content protection system in Embodiment 8.
FIG. 204A is a diagram for describing an example of application to an electronic lock in Embodiment 8.
FIG. 204B is a flowchart of an information communication method in Embodiment 8.
FIG. 204C is a block diagram of an information communication device in Embodiment 8.
FIG. 205 is a diagram for describing an example of application to store visit information transmission in Embodiment 8.
FIG. 206 is a diagram for describing an example of application to location-dependent order control in Embodiment 8.
FIG. 207 is a diagram for describing an example of application to route guidance in Embodiment 8.
FIG. 208 is a diagram for describing an example of application to location notification in Embodiment 8.
FIG. 209 is a diagram for describing an example of application to use log storage and analysis in Embodiment 8.
FIG. 210 is a diagram for describing an example of application to screen sharing in Embodiment 8.
FIG. 211 is a diagram for describing an example of application to screen sharing in Embodiment 8.
FIG. 212 is a diagram for describing an example of application to position estimation using a wireless access point in Embodiment 8.
FIG. 213 is a diagram illustrating a structure of performing position estimation by visible light communication and wireless communication in Embodiment 8.
FIG. 214 is a diagram illustrating an example of application of an information communication method in Embodiment 8.
FIG. 215 is a flowchart illustrating an example of application of an information communication method in Embodiment 8.
FIG. 216 is a flowchart illustrating an example of application of an information communication method in Embodiment 8.
FIG. 217 is a diagram illustrating an example of application of a transmitter and a receiver in Embodiment 9.
FIG. 218 is a diagram illustrating an example of application of a transmitter in Embodiment 9.
FIG. 219 is a flowchart of an information communication method in Embodiment 9.
FIG. 220 is a block diagram of an information communication device in Embodiment 9.
FIG. 221A is a diagram illustrating an example of application of a transmitter and a receiver in Embodiment 9.
FIG. 221B is a flowchart illustrating an example of operation of a receiver in Embodiment 9.
FIG. 222 is a diagram illustrating an example of application of a transmitter and a receiver in Embodiment 9.
FIG. 223 is a diagram illustrating an example of application of a transmitter in Embodiment 9.
FIG. 224A is a diagram illustrating an example of application of a transmitter and a receiver in Embodiment 9.
FIG. 224B is a flowchart illustrating an example of operation of a receiver in Embodiment 9.
FIG. 225 is a diagram illustrating operation of a receiver in Embodiment 9.
FIG. 226 is a diagram illustrating an example of application of a transmitter in Embodiment 9.
FIG. 227 is a diagram illustrating an example of application of a receiver in Embodiment 9.
FIG. 228A is a flowchart illustrating an example of operation of a transmitter in Embodiment 9.
FIG. 228B is a flowchart illustrating an example of operation of a transmitter in Embodiment 9.
FIG. 229 is a flowchart illustrating an example of operation of a transmitter in Embodiment 9.
FIG. 230 is a flowchart illustrating an example of operation of an imaging device in Embodiment 9.
FIG. 231 is a flowchart illustrating an example of operation of an imaging device in Embodiment 9.
FIG. 232 is a diagram illustrating an example of a signal transmitted by a transmitter in Embodiment 9.
FIG. 233 is a diagram illustrating an example of a signal transmitted by a transmitter in Embodiment 9.
FIG. 234 is a diagram illustrating an example of a signal transmitted by a transmitter in Embodiment 9.
FIG. 235 is a diagram illustrating an example of a signal transmitted by a transmitter in Embodiment 9.
FIG. 236 is a diagram illustrating an example of a structure of a system including a transmitter and a receiver in Embodiment 9.
FIG. 237 is a diagram illustrating an example of a structure of a system including a transmitter and a receiver in Embodiment 9.
FIG. 238 is a diagram illustrating an example of a structure of a system including a transmitter and a receiver in Embodiment 9.
FIG. 239 is a diagram illustrating an example of operation of a transmitter in Embodiment 9.
FIG. 240 is a diagram illustrating an example of operation of a transmitter in Embodiment 9.
FIG. 241 is a diagram illustrating an example of operation of a transmitter in Embodiment 9.
FIG. 242 is a diagram illustrating an example of operation of a transmitter in Embodiment 9.
FIG. 243 is a diagram illustrating a watch including light sensors in Embodiment 10.
FIG. 244 is a diagram illustrating an example of a receiver in Embodiment 10.
FIG. 245 is a diagram illustrating an example of a receiver in Embodiment 10.
FIG. 246A is a flowchart of an information communication method according to an aspect of the present disclosure.
FIG. 246B is a block diagram of a mobile terminal according to an aspect of the present disclosure.
FIG. 247 is a diagram illustrating an example of a reception system in Embodiment 10.
FIG. 248 is a diagram illustrating an example of a reception system in Embodiment 10.
FIG. 249A is a diagram illustrating an example of a modulation scheme in Embodiment 10.
FIG. 249B is a diagram illustrating an example of a modulation scheme in Embodiment 10.
FIG. 249C is a diagram illustrating an example of a modulation scheme in Embodiment 10.
FIG. 249D is a diagram illustrating an example of separation of a mixed signal in Embodiment 10.
FIG. 249E is a diagram illustrating an example of separation of a mixed signal in Embodiment 10.
FIG. 249F is a flowchart illustrating processing of an image processing program in Embodiment 10.
FIG. 249G is a block diagram of an information processing apparatus in Embodiment 10.
FIG. 250A is a diagram illustrating an example of a visible light communication system in Embodiment 10.
FIG. 250B is a diagram for describing a use case in Embodiment 10.
FIG. 250C is a diagram illustrating an example of a signal transmission and reception system in Embodiment 10.
FIG. 251 is a flowchart illustrating a reception method in which interference is eliminated in Embodiment 10.
FIG. 252 is a flowchart illustrating a transmitter direction estimation method in Embodiment 10.
FIG. 253 is a flowchart illustrating a reception start method in Embodiment 10.
FIG. 254 is a flowchart illustrating a method of generating an ID additionally using information of another medium in Embodiment 10.
FIG. 255 is a flowchart illustrating a reception scheme selection method by frequency separation in Embodiment 10.
FIG. 256 is a flowchart illustrating a signal reception method in the case of a long exposure time in Embodiment 10.
FIG. 257 is a diagram illustrating an example of a transmitter light adjustment (brightness adjustment) method in Embodiment 10.
FIG. 258 is a diagram illustrating an exemplary method of performing a transmitter light adjustment function in Embodiment 10.
FIG. 259A is a flowchart illustrating an example of operation of a receiver in Embodiment 11.
FIG. 259B is a flowchart illustrating an example of operation of a receiver in Embodiment 11.
FIG. 259C is a flowchart illustrating an example of operation of a receiver in Embodiment 11.
FIG. 259D is a flowchart illustrating an example of operation of a receiver in Embodiment 11.
FIG. 260 is a diagram for describing EX zoom.
FIG. 261A is a flowchart illustrating processing of a reception program in Embodiment 10.
FIG. 261B is a block diagram of a reception device in Embodiment 10.
FIG. 262 is a diagram illustrating an example of a signal reception method in Embodiment 12.
FIG. 263 is a diagram illustrating an example of a signal reception method in Embodiment 12.
FIG. 264 is a diagram illustrating an example of a signal reception method in Embodiment 12.
FIG. 265 is a diagram illustrating an example of a screen display method used by a receiver in Embodiment 12.
FIG. 266 is a diagram illustrating an example of a signal reception method in Embodiment 12.
FIG. 267 is a diagram illustrating an example of a signal reception method in Embodiment 12.
FIG. 268 is a flowchart illustrating an example of a signal reception method in Embodiment 12.
FIG. 269 is a diagram illustrating an example of a signal reception method in Embodiment 12.
FIG. 270A is a flowchart illustrating processing of a reception program in Embodiment 12.
FIG. 270B is a block diagram of a reception device in Embodiment 12.
FIG. 271 is a diagram illustrating an example of what is displayed on a receiver when a visible light signal is received.
FIG. 272 is a diagram illustrating an example of what is displayed on a receiver when a visible light signal is received.
FIG. 273 is a diagram illustrating a display example of obtained data image.
FIG. 274 is a diagram illustrating an operation example for storing or discarding obtained data.
FIG. 275 is a diagram illustrating an example of what is displayed when obtained data is browsed.
FIG. 276 is a diagram illustrating an example of a transmitter in Embodiment 12.
FIG. 277 is a diagram illustrating an example of a reception method in Embodiment 12.
FIG. 278 is a diagram illustrating an example of a header pattern in Embodiment 13.
FIG. 279 is a diagram for describing an example of a packet structure in a communication protocol in Embodiment 13.
FIG. 280 is a flowchart illustrating an example of a reception method in Embodiment 13.
FIG. 281 is a flowchart illustrating an example of a reception method in Embodiment 13.
FIG. 282 is a flowchart illustrating an example of a reception method in Embodiment 13.
FIG. 283 is a diagram for describing a reception method in which a receiver in Embodiment 13 uses a exposure time longer than a period of a modulation frequency (a modulation period).
FIG. 284 is a diagram for describing a reception method in which a receiver in Embodiment 13 uses a exposure time longer than a period of a modulation frequency (a modulation period).
FIG. 285 is a diagram indicating an efficient number of divisions relative to a size of transmission data in Embodiment 13.
FIG. 286A is a diagram illustrating an example of a setting method in Embodiment 13.
FIG. 286B is a diagram illustrating another example of a setting method in Embodiment 13.
FIG. 287A is a flowchart illustrating processing of an image processing program in Embodiment 13.
FIG. 287B is a block diagram of an information processing apparatus in Embodiment 13.
FIG. 288 is a diagram for describing an example of application of a transmission and reception system in Embodiment 13.
FIG. 289 is a flowchart illustrating processing operation of a transmission and reception system in Embodiment 13.
FIG. 290 is a diagram for describing an example of application of a transmission and reception system in Embodiment 13.
FIG. 291 is a flowchart illustrating processing operation of a transmission and reception system in Embodiment 13.
FIG. 292 is a diagram for describing an example of application of a transmission and reception system in Embodiment 13.
FIG. 293 is a flowchart illustrating processing operation of a transmission and reception system in Embodiment 13.
FIG. 294 is a diagram for describing an example of application of a transmitter in Embodiment 13.
FIG. 295 is a diagram for describing an example of application of a transmission and reception system in Embodiment 14.
FIG. 296 is a diagram for describing an example of application of a transmission and reception system in Embodiment 14.
FIG. 297 is a diagram for describing an example of application of a transmission and reception system in Embodiment 14.
FIG. 298 is a diagram for describing an example of application of a transmission and reception system in Embodiment 14.
FIG. 299 is a diagram for describing an example of application of a transmission and reception system in Embodiment 14.
FIG. 300 is a diagram for describing an example of application of a transmission and reception system in Embodiment 14.
FIG. 301 is a diagram for describing an example of application of a transmission and reception system in Embodiment 14.
FIG. 302 is a diagram for describing an example of application of a transmission and reception system in Embodiment 14.
FIG. 303 is a diagram for describing an example of application of a transmission and reception system in Embodiment 14.
FIG. 304 is a diagram for describing an example of application of a transmission and reception system in Embodiment 14.
FIG. 305 is a diagram for describing an example of application of a transmission and reception system in Embodiment 14.
FIG. 306 is a diagram for describing an example of application of a transmission and reception system in Embodiment 14.
FIG. 307 is a diagram for describing an example of application of a transmission and reception system in Embodiment 14.
FIG. 308 is a diagram for describing an example of application of a transmission and reception system in Embodiment 14.
FIG. 309A is a diagram for describing a transmitter in Embodiment 15.
FIG. 309B is a diagram illustrating a change in luminance of each of R, G, and B in Embodiment 15.
FIG. 310 is a diagram illustrating persistence properties of a green phosphorus element and a red phosphorus element in Embodiment 15.
FIG. 311 is a diagram for explaining a new problem that will occur in an attempt to reduce errors in reading a barcode in Embodiment 15.
FIG. 312 is a diagram for describing downsampling performed by a receiver in Embodiment 15.
FIG. 313 is a flowchart illustrating processing operation of a receiver in Embodiment 15.
FIG. 314 is a diagram illustrating processing operation of a reception device (an imaging device) in Embodiment 16.
FIG. 315 is a diagram illustrating processing operation of a reception device (an imaging device) in Embodiment 16.
FIG. 316 is a diagram illustrating processing operation of a reception device (an imaging device) in Embodiment 16.
FIG. 317 is a diagram illustrating processing operation of a reception device (an imaging device) in Embodiment 16.
FIG. 318 is a diagram illustrating an example of a transmission signal in Embodiment 17.
FIG. 319 is a diagram illustrating an example of a transmission signal in Embodiment 17.
FIG. 320 is a diagram illustrating an example of a transmission signal in Embodiment 17.
FIG. 321 is a diagram illustrating an example of a transmission signal in Embodiment 17.
FIG. 322 is a diagram illustrating an example of a transmission signal in Embodiment 17.
FIG. 323 is a diagram illustrating an example of a transmission signal in Embodiment 17.
FIG. 324 is a diagram illustrating an example of a transmission signal in Embodiment 17.
FIG. 325 is a diagram illustrating an example of a reception algorithm in Embodiment 17.
FIG. 326 is a diagram illustrating an example of a reception algorithm in Embodiment 17.
FIG. 327 is a diagram illustrating an example of a reception algorithm in Embodiment 17.
FIG. 328 is a diagram illustrating an example of a reception algorithm in Embodiment 17.
An information processing program according to an aspect of the present disclosure is an information processing program for causing a computer to process information to be transmitted, in order for the information to be transmitted by way of luminance change, and causes the computer to execute: encoding the information to determine a luminance change frequency; and outputting a signal of the luminance change frequency determined, to cause a light emitter to change in luminance according to the luminance change frequency determined, to transmit the information, wherein in the encoding, each of a first frequency and a second frequency different from the first frequency is determined as the luminance change frequency, and in the outputting, each of a signal of the first frequency and a signal of the second frequency is output as the signal of the luminance change frequency determined, to cause the light emitter to change in luminance according to the first frequency during a first time and change in luminance according to the second frequency during a second time after the first time elapses, the second time being different from the first time. For example, the first time is a duration corresponding to one period of the first frequency, and the second time is a duration corresponding to one period of the second frequency.
With this, information to be transmitted can be appropriately transmitted in the form of visible light signals of the first and second frequencies as illustrated in FIGS. 249A to 249G. Furthermore, with the first time and the second time being different, the transmission can be adapted to various situations. As a result, communication between various devices becomes possible.
Furthermore, in the outputting, at least one of the signal of the first frequency and the signal of the second frequency may be repeatedly output to make a total number of times the signal of the first frequency is output and a total number of times the signal of the second frequency is output different from each other.
With this, the transmission can be adapted to various situations.
Furthermore, in the outputting, at least one of the signal of the first frequency and the signal of the second frequency may be repeatedly output to make a total number of times one of the signal of the first frequency and the signal of the second frequency that has a lower frequency is output, greater than a total number of times a remaining one of the signal of the first frequency and the signal of the second frequency that has a higher frequency is output.
With this, in the case where the light emitter changes in luminance according to the frequency specified by each output signal, the light emitter can transmit, with high luminance, the information to be transmitted. For example, suppose the duration for which low luminance lasts is the same in the change in luminance according to a low frequency, namely, the first frequency, and the change in luminance according to a high frequency, namely, the second frequency. In this case, the duration for which high luminance lasts is longer in the change in luminance according to the first frequency (that is, a low frequency) than in the change in luminance according to the second frequency (that is, a high frequency). Therefore, when many signals of the first frequency are output, the light emitter can transmit, with high luminance, the information to be transmitted.
Furthermore, in the outputting, at least one of the signal of the first frequency and the signal of the second frequency may be repeatedly output to make a total number of times one of the signal of the first frequency and the signal of the second frequency that has a higher frequency is output, greater than a total number of times a remaining one of the signal of the first frequency and the signal of the second frequency that has a lower frequency is output.
With this, in the case where the light emitter changes in luminance according to the frequency specified by each output signal, the reception efficiency of the information to be transmitted by such a change in luminance can be higher. For example, when the information to be transmitted is transmitted to the receiver in the form of visible light signals represented by a plurality of frequencies, the receiver performs frequency analysis, such as the Fourier transform, on a captured image, to detect a frequency peak included in the visible light signal. Here, with a higher frequency, such peak detection is more difficult. Therefore, the signal of the first frequency and the signal of the second frequency are output so that the number of times one of the signals having a higher frequency is output becomes greater than the number of times one of the signals having a lower frequency is output as described above, so that the peak detection with a high frequency can be facilitated. As a result, the reception efficiency can be improved.
Furthermore, in the outputting, at least one of the signal of the first frequency and the signal of the second frequency may be repeatedly output to avoid continuous output of a signal of a same frequency.
With this, in the case where the light emitter changes in luminance according to the frequency specified by each output signal, it can make it harder for human eyes or cameras to catch flicker of light from the light emitter.
Furthermore, a reception program for receiving information from a light emitter changing in luminance according to a signal output using the above image processing program may cause a computer to execute: setting a first exposure time for a plurality of imaging elements which are a part of K imaging elements included in an image sensor, and setting a second exposure time for a plurality of imaging elements which are a remainder of the K imaging elements, where K is an integer of 4 or more, the second exposure time being shorter than the first exposure time; causing the image sensor to capture a subject with the set first exposure time and the set second exposure time, to obtain a normal image according to output from the plurality of the imaging elements for which the first exposure time is set, and obtain a bright line image according to output from the plurality of the imaging elements for which the second exposure time is set, the subject being the light emitter changing in luminance, the bright line image including a plurality of bright lines each of which corresponds to a different one of a plurality of exposure lines included in the image sensor; and obtaining the information by decoding a pattern of the plurality of the bright lines included in the obtained bright line image.
With this, imaging is performed by the plurality of the imaging elements for which the first exposure time is set and the plurality of the imaging elements for which the second exposure time is set as illustrated in FIGS. 262 to 270B, with the result that a normal image and a bright line image can be obtained in a single imaging operation by the image sensor. That is, it is possible to capture a normal image and obtain information by visible light communication at the same time.
Furthermore, it may be that in the setting, the first exposure time is set for a plurality of imaging element lines which are a part of L imaging element lines included in the image sensor where L is an integer of 4 or more, the second exposure time is set for a plurality of imaging element lines which are a remainder of the L imaging element lines, and each of the L imaging element lines includes a plurality of imaging elements included in the image sensor and arranged in a line. For example, each of the L imaging element lines is a different one of the plurality of the exposure lines included in the image sensor. Alternatively, each of the L imaging element lines includes a plurality of imaging elements included in the image sensor and arranged along a direction perpendicular to the plurality of the exposure lines.
With this, it is possible to set an exposure time for each imaging element line, which is a large unit, without individually setting an exposure time for each imaging element, which is a small unit, so that the processing load can be reduced.
Furthermore, it may be that in the setting, one of the first exposure time and the second exposure time is set for each of odd-numbered imaging element lines of the L imaging element lines included in the image sensor, to set a same exposure time for each of the odd-numbered imaging element lines, and a remaining one of the first exposure time and the second exposure time is set for each of even-numbered imaging element lines of the L imaging element lines, to set a same exposure time for each of the even-numbered imaging element lines, and when the setting, the causing, and the obtaining are repeated, in a current round of the setting, an exposure time for each of the odd-numbered imaging element lines is set to an exposure time set for each of the even-numbered imaging element lines in an immediately previous round of the setting, and an exposure time for each of the even-numbered imaging element lines is set to an exposure time set for each of the odd-numbered imaging element lines in the immediately previous round of the setting.
With this, at every operation to obtain a normal image, the plurality of the imaging element lines that are to be used in the obtainment can be switched between odd-numbered imaging element lines and even-numbered imaging element lines. As a result, each of the sequentially obtained normal images can be displayed in an interlaced format. Furthermore, by interpolating two continuously obtained normal images with each other, it is possible to generate a new normal image that includes an image obtained by the odd-numbered imaging element lines and the even-numbered imaging element lines.
Furthermore, it may be that in the setting, a preset mode is switched between a normal imaging priority mode and a visible light imaging priority mode, when the preset mode is switched to the normal imaging priority mode, a total number of the imaging elements for which the first exposure time is set is greater than a total number of the imaging elements for which the second exposure time is set, and when the preset mode is switched to the visible light imaging priority mode, a total number of the imaging elements for which the first exposure time is set is less than a total number of the imaging elements for which the second exposure time is set.
With this, when the preset mode is switched to the normal imaging priority mode, the quality of the normal image can be increased, and when the preset mode is switched to the visible light imaging priority mode, the reception efficiency for information from the light emitter can be increased.
Furthermore, in the setting, an exposure time may be set for each imaging element included in the image sensor, to distribute, in a checkered pattern, the plurality of the imaging elements for which the first exposure time is set and the plurality of the imaging elements for which the second exposure time is set.
This results in uniform distribution of the plurality of the imaging elements for which the first exposure time is set and the plurality of the imaging elements for which the second exposure time is set, so that it is possible to obtain the normal image and the bright line image, the quality of which is not unbalanced between the horizontal direction and the vertical direction.
These general and specific aspects may be implemented using an apparatus, a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or any combination of apparatuses, systems, methods, integrated circuits, computer programs, or computer-readable recording media.
Each of the embodiments described below shows a general or specific example.
Each of the embodiments described below shows a general or specific example. The numerical values, shapes, materials, structural elements, the arrangement and connection of the structural elements, steps, the processing order of the steps etc. shown in the following embodiments are mere examples, and therefore do not limit the scope of the present disclosure. Therefore, among the structural elements in the following embodiments, structural elements not recited in any one of the independent claims representing the broadest concepts are described as arbitrary structural elements.
The following describes Embodiment 1.
The following proposes an imaging method in which, when capturing one image, all imaging elements are not exposed simultaneously but the times of starting and ending the exposure differ between the imaging elements. FIG. 1 illustrates an example of imaging where imaging elements arranged in a line are exposed simultaneously, with the exposure start time being shifted in order of lines. Here, the simultaneously exposed imaging elements are referred to as “exposure line”, and the line of pixels in the image corresponding to the imaging elements is referred to as “bright line”.
In the case of capturing a blinking light source shown on the entire imaging elements using this imaging method, bright lines (lines of brightness in pixel value) along exposure lines appear in the captured image as illustrated in FIG. 2. By recognizing this bright line pattern, the luminance change of the light source at a speed higher than the imaging frame rate can be estimated. Hence, transmitting a signal as the luminance change of the light source enables communication at a speed not less than the imaging frame rate. In the case where the light source takes two luminance values to express a signal, the lower luminance value is referred to as “low” (LO), and the higher luminance value is referred to as “high” (HI). The low may be a state in which the light source emits no light, or a state in which the light source emits weaker light than in the high.
By this method, information transmission is performed at a speed higher than the imaging frame rate.
In the case where the number of exposure lines whose exposure times do not overlap each other is 20 in one captured image and the imaging frame rate is 30 fps, it is possible to recognize a luminance change in a period of 1.67 millisecond. In the case where the number of exposure lines whose exposure times do not overlap each other is 1000, it is possible to recognize a luminance change in a period of 1/30000 second (about 33 microseconds). Note that the exposure time is set to less than 10 milliseconds, for example.
FIG. 2 illustrates a situation where, after the exposure of one exposure line ends, the exposure of the next exposure line starts.
In this situation, when transmitting information based on whether or not each exposure line receives at least a predetermined amount of light, information transmission at a speed of fl bits per second at the maximum can be realized where f is the number of frames per second (frame rate) and l is the number of exposure lines constituting one image.
Note that faster communication is possible in the case of performing time-difference exposure not on a line basis but on a pixel basis.
In such a case, when transmitting information based on whether or not each pixel receives at least a predetermined amount of light, the transmission speed is firm bits per second at the maximum, where m is the number of pixels per exposure line.
If the exposure state of each exposure line caused by the light emission of the light emitting unit is recognizable in a plurality of levels as illustrated in FIG. 3, more information can be transmitted by controlling the light emission time of the light emitting unit in a shorter unit of time than the exposure time of each exposure line.
In the case where the exposure state is recognizable in Elv levels, information can be transmitted at a speed of flElv bits per second at the maximum.
Moreover, a fundamental period of transmission can be recognized by causing the light emitting unit to emit light with a timing slightly different from the timing of exposure of each exposure line.
FIG. 4 illustrates a situation where, before the exposure of one exposure line ends, the exposure of the next exposure line starts. That is, the exposure times of adjacent exposure lines partially overlap each other. This structure has the feature (1): the number of samples in a predetermined time can be increased as compared with the case where, after the exposure of one exposure line ends, the exposure of the next exposure line starts. The increase of the number of samples in the predetermined time leads to more appropriate detection of the light signal emitted from the light transmitter which is the subject. In other words, the error rate when detecting the light signal can be reduced. The structure also has the feature (2): the exposure time of each exposure line can be increased as compared with the case where, after the exposure of one exposure line ends, the exposure of the next exposure line starts. Accordingly, even in the case where the subject is dark, a brighter image can be obtained, i.e. the S/N ratio can be improved. Here, the structure in which the exposure times of adjacent exposure lines partially overlap each other does not need to be applied to all exposure lines, and part of the exposure lines may not have the structure of partially overlapping in exposure time. By keeping part of the exposure lines from partially overlapping in exposure time, the occurrence of an intermediate color caused by exposure time overlap is suppressed on the imaging screen, as a result of which bright lines can be detected more appropriately.
In this situation, the exposure time is calculated from the brightness of each exposure line, to recognize the light emission state of the light emitting unit.
Note that, in the case of determining the brightness of each exposure line in a binary fashion of whether or not the luminance is greater than or equal to a threshold, it is necessary for the light emitting unit to continue the state of emitting no light for at least the exposure time of each line, to enable the no light emission state to be recognized.
FIG. 5A illustrates the influence of the difference in exposure time in the case where the exposure start time of each exposure line is the same. In 7500 a, the exposure end time of one exposure line and the exposure start time of the next exposure line are the same. In 7500 b, the exposure time is longer than that in 7500 a. The structure in which the exposure times of adjacent exposure lines partially overlap each other as in 7500 b allows a longer exposure time to be used. That is, more light enters the imaging element, so that a brighter image can be obtained. In addition, since the imaging sensitivity for capturing an image of the same brightness can be reduced, an image with less noise can be obtained. Communication errors are prevented in this way.
FIG. 5B illustrates the influence of the difference in exposure start time of each exposure line in the case where the exposure time is the same. In 7501 a, the exposure end time of one exposure line and the exposure start time of the next exposure line are the same. In 7501 b, the exposure of one exposure line ends after the exposure of the next exposure line starts. The structure in which the exposure times of adjacent exposure lines partially overlap each other as in 7501 b allows more lines to be exposed per unit time. This increases the resolution, so that more information can be obtained. Since the sample interval (i.e. the difference in exposure start time) is shorter, the luminance change of the light source can be estimated more accurately, contributing to a lower error rate. Moreover, the luminance change of the light source in a shorter time can be recognized. By exposure time overlap, light source blinking shorter than the exposure time can be recognized using the difference of the amount of exposure between adjacent exposure lines.
As described with reference to FIGS. 5A and 5B, in the structure in which each exposure line is sequentially exposed so that the exposure times of adjacent exposure lines partially overlap each other, the communication speed can be dramatically improved by using, for signal transmission, the bright line pattern generated by setting the exposure time shorter than in the normal imaging mode. Setting the exposure time in visible light communication to less than or equal to 1/480 second enables an appropriate bright line pattern to be generated. Here, it is necessary to set (exposure time)<⅛×f, where f is the frame frequency. Blanking during imaging is half of one frame at the maximum. That is, the blanking time is less than or equal to half of the imaging time. The actual imaging time is therefore ½f at the shortest. Besides, since 4-value information needs to be received within the time of ½f, it is necessary to at least set the exposure time to less than 1/(2f×4). Given that the normal frame rate is less than or equal to 60 frames per second, by setting the exposure time to less than or equal to 1/480 second, an appropriate bright line pattern is generated in the image data and thus fast signal transmission is achieved.
FIG. 5C illustrates the advantage of using a short exposure time in the case where each exposure line does not overlap in exposure time. In the case where the exposure time is long, even when the light source changes in luminance in a binary fashion as in 7502 a, an intermediate-color part tends to appear in the captured image as in 7502 e, making it difficult to recognize the luminance change of the light source. By providing a predetermined non-exposure blank time (predetermined wait time) tD2 from when the exposure of one exposure line ends to when the exposure of the next exposure line starts as in 7502 d, however, the luminance change of the light source can be recognized more easily. That is, a more appropriate bright line pattern can be detected as in 7502 f. The provision of the predetermined non-exposure blank time is possible by setting a shorter exposure time tE than the time difference tD between the exposure start times of the exposure lines, as in 7502 d. In the case where the exposure times of adjacent exposure lines partially overlap each other in the normal imaging mode, the exposure time is shortened from the normal imaging mode so as to provide the predetermined non-exposure blank time. In the case where the exposure end time of one exposure line and the exposure start time of the next exposure line are the same in the normal imaging mode, too, the exposure time is shortened so as to provide the predetermined non-exposure time. Alternatively, the predetermined non-exposure blank time (predetermined wait time) tD2 from when the exposure of one exposure line ends to when the exposure of the next exposure line starts may be provided by increasing the interval tD between the exposure start times of the exposure lines, as in 7502 g. This structure allows a longer exposure time to be used, so that a brighter image can be captured. Moreover, a reduction in noise contributes to higher error tolerance. Meanwhile, this structure is disadvantageous in that the number of samples is small as in 7502 h, because fewer exposure lines can be exposed in a predetermined time. Accordingly, it is desirable to use these structures depending on circumstances. For example, the estimation error of the luminance change of the light source can be reduced by using the former structure in the case where the imaging object is bright and using the latter structure in the case where the imaging object is dark.
Here, the structure in which the exposure times of adjacent exposure lines partially overlap each other does not need to be applied to all exposure lines, and part of the exposure lines may not have the structure of partially overlapping in exposure time. Moreover, the structure in which the predetermined non-exposure blank time (predetermined wait time) is provided from when the exposure of one exposure line ends to when the exposure of the next exposure line starts does not need to be applied to all exposure lines, and part of the exposure lines may have the structure of partially overlapping in exposure time. This makes it possible to take advantage of each of the structures. Furthermore, the same reading method or circuit may be used to read a signal in the normal imaging mode in which imaging is performed at the normal frame rate (30 fps, 60 fps) and the visible light communication mode in which imaging is performed with the exposure time less than or equal to 1/480 second for visible light communication. The use of the same reading method or circuit to read a signal eliminates the need to employ separate circuits for the normal imaging mode and the visible light communication mode. The circuit size can be reduced in this way.
FIG. 5D illustrates the relation between the minimum change time tS of light source luminance, the exposure time tE, the time difference tD between the exposure start times of the exposure lines, and the captured image. In the case where tE+tD<tS, imaging is always performed in a state where the light source does not change from the start to end of the exposure of at least one exposure line. As a result, an image with clear luminance is obtained as in 7503 d, from which the luminance change of the light source is easily recognizable. In the case where 2tE>tS, a bright line pattern different from the luminance change of the light source might be obtained, making it difficult to recognize the luminance change of the light source from the captured image.
FIG. 5E illustrates the relation between the transition time tT of light source luminance and the time difference tD between the exposure start times of the exposure lines. When tD is large as compared with tT, fewer exposure lines are in the intermediate color, which facilitates estimation of light source luminance. It is desirable that tD>tT, because the number of exposure lines in the intermediate color is two or less consecutively. Since tT is less than or equal to 1 microsecond in the case where the light source is an LED and about 5 microseconds in the case where the light source is an organic EL device, setting tD to greater than or equal to 5 microseconds facilitates estimation of light source luminance.
FIG. 5F illustrates the relation between the high frequency noise tHT of light source luminance and the exposure time tE. When tE is large as compared with tHT, the captured image is less influenced by high frequency noise, which facilitates estimation of light source luminance. When tE is an integral multiple of tHT, there is no influence of high frequency noise, and estimation of light source luminance is easiest. For estimation of light source luminance, it is desirable that tE>tHT. High frequency noise is mainly caused by a switching power supply circuit. Since tHT is less than or equal to 20 microseconds in many switching power supplies for lightings, setting tE to greater than or equal to 20 microseconds facilitates estimation of light source luminance.
FIG. 5G is a graph representing the relation between the exposure time tE and the magnitude of high frequency noise when tHT is 20 microseconds. Given that tHT varies depending on the light source, the graph demonstrates that it is efficient to set tE to greater than or equal to 15 microseconds, greater than or equal to 35 microseconds, greater than or equal to 54 microseconds, or greater than or equal to 74 microseconds, each of which is a value equal to the value when the amount of noise is at the maximum. Though tE is desirably larger in terms of high frequency noise reduction, there is also the above-mentioned property that, when tE is smaller, an intermediate-color part is less likely to occur and estimation of light source luminance is easier. Therefore, tE may be set to greater than or equal to 15 microseconds when the light source luminance change period is 15 to 35 microseconds, to greater than or equal to 35 microseconds when the light source luminance change period is 35 to 54 microseconds, to greater than or equal to 54 microseconds when the light source luminance change period is 54 to 74 microseconds, and to greater than or equal to 74 microseconds when the light source luminance change period is greater than or equal to 74 microseconds.
FIG. 5H illustrates the relation between the exposure time tE and the recognition success rate. Since the exposure time tE is relative to the time during which the light source luminance is constant, the horizontal axis represents the value (relative exposure time) obtained by dividing the light source luminance change period tS by the exposure time tE. It can be understood from the graph that the recognition success rate of approximately 100% can be attained by setting the relative exposure time to less than or equal to 1.2. For example, the exposure time may be set to less than or equal to approximately 0.83 millisecond in the case where the transmission signal is 1 kHz. Likewise, the recognition success rate greater than or equal to 95% can be attained by setting the relative exposure time to less than or equal to 1.25, and the recognition success rate greater than or equal to 80% can be attained by setting the relative exposure time to less than or equal to 1.4. Moreover, since the recognition success rate sharply decreases when the relative exposure time is about 1.5 and becomes roughly 0% when the relative exposure time is 1.6, it is necessary to set the relative exposure time not to exceed 1.5. After the recognition rate becomes 0% at 7507 c, it increases again at 7507 d, 7507 e, and 7507 f. Accordingly, for example to capture a bright image with a longer exposure time, the exposure time may be set so that the relative exposure time is 1.9 to 2.2, 2.4 to 2.6, or 2.8 to 3.0. Such an exposure time may be used, for instance, as an intermediate mode in FIG. 7.
FIG. 6A is a flowchart of an information communication method in this embodiment.
The information communication method in this embodiment is an information communication method of obtaining information from a subject, and includes Steps SK91 to SK93.
In detail, the information communication method includes: a first exposure time setting step SK91 of setting a first exposure time of an image sensor so that, in an image obtained by capturing the subject by the image sensor, a plurality of bright lines corresponding to a plurality of exposure lines included in the image sensor appear according to a change in luminance of the subject; a first image obtainment step SK92 of obtaining a bright line image including the plurality of bright lines, by capturing the subject changing in luminance by the image sensor with the set first exposure time; and an information obtainment step SK93 of obtaining the information by demodulating data specified by a pattern of the plurality of bright lines included in the obtained bright line image, wherein in the first image obtainment step SK92, exposure starts sequentially for the plurality of exposure lines each at a different time, and exposure of each of the plurality of exposure lines starts after a predetermined blank time elapses from when exposure of an adjacent exposure line adjacent to the exposure line ends.
FIG. 6B is a block diagram of an information communication device in this embodiment.
An information communication device K90 in this embodiment is an information communication device that obtains information from a subject, and includes structural elements K91 to K93.
In detail, the information communication device K90 includes: an exposure time setting unit K91 that sets an exposure time of an image sensor so that, in an image obtained by capturing the subject by the image sensor, a plurality of bright lines corresponding to a plurality of exposure lines included in the image sensor appear according to a change in luminance of the subject; an image obtainment unit K92 that includes the image sensor, and obtains a bright line image including the plurality of bright lines by capturing the subject changing in luminance with the set exposure time; and an information obtainment unit K93 that obtains the information by demodulating data specified by a pattern of the plurality of bright lines included in the obtained bright line image, wherein exposure starts sequentially for the plurality of exposure lines each at a different time, and exposure of each of the plurality of exposure lines starts after a predetermined blank time elapses from when exposure of an adjacent exposure line adjacent to the exposure line ends.
In the information communication method and the information communication device K90 illustrated in FIGS. 6A and 6B, the exposure of each of the plurality of exposure lines starts a predetermined blank time after the exposure of the adjacent exposure line adjacent to the exposure line ends, for instance as illustrated in FIG. 5C. This eases the recognition of the change in luminance of the subject. As a result, the information can be appropriately obtained from the subject.
It should be noted that in the above embodiment, each of the constituent elements may be constituted by dedicated hardware, or may be obtained by executing a software program suitable for the constituent element. Each constituent element may be achieved by a program execution unit such as a CPU or a processor reading and executing a software program stored in a recording medium such as a hard disk or semiconductor memory. For example, the program causes a computer to execute the information communication method illustrated in the flowchart of FIG. 6A.
This embodiment describes each example of application using a receiver such as a smartphone which is the information communication device D90 and a transmitter for transmitting information as a blink pattern of the light source such as an LED or an organic EL device in Embodiment 1 described above.
FIG. 7 is a diagram illustrating an example of each mode of a receiver in this embodiment.
In the normal imaging mode, a receiver 8000 performs imaging at a shutter speed of 1/100 second as an example to obtain a normal captured image, and displays the normal captured image on a display. For example, a subject such as a street lighting or a signage as a store sign and its surroundings are clearly shown in the normal captured image.
In the visible light communication mode, the receiver 8000 performs imaging at a shutter speed of 1/10000 second as an example, to obtain a visible light communication image. For example, in the case where the above-mentioned street lighting or signage is transmitting a signal by way of luminance change as the light source described in Embodiment 1, that is, a transmitter, one or more bright lines (hereafter referred to as “bright line pattern”) are shown in the signal transmission part of the visible light communication image, while nothing is shown in the other part. That is, in the visible light communication image, only the bright line pattern is shown and the part of the subject not changing in luminance and the surroundings of the subject are not shown.
In the intermediate mode, the receiver 8000 performs imaging at a shutter speed of 1/3000 second as an example, to obtain an intermediate image. In the intermediate image, the bright line pattern is shown, and the part of the subject not changing in luminance and the surroundings of the subject are shown, too. By the receiver 8000 displaying the intermediate image on the display, the user can find out from where or from which position the signal is being transmitted. Note that the bright line pattern, the subject, and its surroundings shown in the intermediate image are not as clear as the bright line pattern in the visible light communication image and the subject and its surroundings in the normal captured image respectively, but have the level of clarity recognizable by the user.
In the following description, the normal imaging mode or imaging in the normal imaging mode is referred to as “normal imaging”, and the visible light communication mode or imaging in the visible light communication mode is referred to as “visible light imaging” (visible light communication). Imaging in the intermediate mode may be used instead of normal imaging and visible light imaging, and the intermediate image may be used instead of the below-mentioned synthetic image.
FIG. 8 is a diagram illustrating an example of imaging operation of a receiver in this embodiment.
The receiver 8000 switches the imaging mode in such a manner as normal imaging, visible light communication, normal imaging, . . . . The receiver 8000 synthesizes the normal captured image and the visible light communication image to generate a synthetic image in which the bright line pattern, the subject, and its surroundings are clearly shown, and displays the synthetic image on the display. The synthetic image is an image generated by superimposing the bright line pattern of the visible light communication image on the signal transmission part of the normal captured image. The bright line pattern, the subject, and its surroundings shown in the synthetic image are clear, and have the level of clarity sufficiently recognizable by the user. Displaying such a synthetic image enables the user to more distinctly find out from which position the signal is being transmitted.
FIG. 9 is a diagram illustrating another example of imaging operation of a receiver in this embodiment.
The receiver 8000 includes a camera Ca1 and a camera Ca2. In the receiver 8000, the camera Ca1 performs normal imaging, and the camera Ca2 performs visible light imaging. Thus, the camera Ca1 obtains the above-mentioned normal captured image, and the camera Ca2 obtains the above-mentioned visible light communication image. The receiver 8000 synthesizes the normal captured image and the visible light communication image to generate the above-mentioned synthetic image, and displays the synthetic image on the display.
FIG. 10A is a diagram illustrating another example of imaging operation of a receiver in this embodiment.
In the receiver 8000 including two cameras, the camera Ca1 switches the imaging mode in such a manner as normal imaging, visible light communication, normal imaging, . . . . Meanwhile, the camera Ca2 continuously performs normal imaging. When normal imaging is being performed by the cameras Ca1 and Ca2 simultaneously, the receiver 8000 estimates the distance (hereafter referred to as “subject distance”) from the receiver 8000 to the subject based on the normal captured images obtained by these cameras, through the use of stereoscopy (triangulation principle). By using such estimated subject distance, the receiver 8000 can superimpose the bright line pattern of the visible light communication image on the normal captured image at the appropriate position. The appropriate synthetic image can be generated in this way.
FIG. 10B is a diagram illustrating another example of imaging operation of a receiver in this embodiment.
The receiver 8000 includes three cameras (cameras Ca1, Ca2, and Ca3) as an example. In the receiver 8000, two cameras (cameras Ca2 and Ca3) continuously perform normal imaging, and the remaining camera (camera Ca1) continuously performs visible light communication. Hence, the subject distance can be estimated at any timing, based on the normal captured images obtained by two cameras engaged in normal imaging.
FIG. 10C is a diagram illustrating another example of imaging operation of a receiver in this embodiment.
The receiver 8000 includes three cameras (cameras Ca1, Ca2, and Ca3) as an example. In the receiver 8000, each camera switches the imaging mode in such a manner as normal imaging, visible light communication, normal imaging, . . . . The imaging mode of each camera is switched per period so that, in one period, two cameras perform normal imaging and the remaining camera performs visible light communication. That is, the combination of cameras engaged in normal imaging is changed periodically. Hence, the subject distance can be estimated in any period, based on the normal captured images obtained by two cameras engaged in normal imaging.
FIG. 11A is a diagram illustrating an example of camera arrangement of a receiver in this embodiment.
In the case where the receiver 8000 includes two cameras Ca1 and Ca2, the two cameras Ca1 and Ca2 are positioned away from each other as illustrated in FIG. 11A. The subject distance can be accurately estimated in this way. In other words, the subject distance can be estimated more accurately when the distance between two cameras is longer.
FIG. 11B is a diagram illustrating another example of camera arrangement of a receiver in this embodiment.
In the case where the receiver 8000 includes three cameras Ca1, Ca2, and Ca3, the two cameras Ca1 and Ca2 for normal imaging are positioned away from each other as illustrated in FIG. 11B, and the camera Ca3 for visible light communication is, for example, positioned between the cameras Ca1 and Ca2. The subject distance can be accurately estimated in this way. In other words, the subject distance can be accurately estimated by using two farthest cameras for normal imaging.
FIG. 12 is a diagram illustrating an example of display operation of a receiver in this embodiment.
The receiver 8000 switches the imaging mode in such a manner as visible light communication, normal imaging, visible light communication, . . . , as mentioned above. Upon performing visible light communication first, the receiver 8000 starts an application program. The receiver 8000 then estimates its position based on the signal received by visible light communication. Next, when performing normal imaging, the receiver 8000 displays AR (Augmented Reality) information on the normal captured image obtained by normal imaging. The AR information is obtained based on, for example, the position estimated as mentioned above. The receiver 8000 also estimates the change in movement and direction of the receiver 8000 based on the detection result of the 9-axis sensor, the motion detection in the normal captured image, and the like, and moves the display position of the AR information according to the estimated change in movement and direction. This enables the AR information to follow the subject image in the normal captured image.
When switching the imaging mode from normal imaging to visible light communication, in visible light communication the receiver 8000 superimposes the AR information on the latest normal captured image obtained in immediately previous normal imaging. The receiver 8000 then displays the normal captured image on which the AR information is superimposed. The receiver 8000 also estimates the change in movement and direction of the receiver 8000 based on the detection result of the 9-axis sensor, and moves the AR information and the normal captured image according to the estimated change in movement and direction, in the same way as in normal imaging. This enables the AR information to follow the subject image in the normal captured image according to the movement of the receiver 8000 and the like in visible light communication, as in normal imaging. Moreover, the normal image can be enlarged or reduced according to the movement of the receiver 8000 and the like.
FIG. 13 is a diagram illustrating an example of display operation of a receiver in this embodiment.
For example, the receiver 8000 may display the synthetic image in which the bright line pattern is shown, as illustrated in (a) in FIG. 13. As an alternative, the receiver 8000 may superimpose, instead of the bright line pattern, a signal specification object which is an image having a predetermined color for notifying signal transmission on the normal captured image to generate the synthetic image, and display the synthetic image, as illustrated in (b) in FIG. 13.
As another alternative, the receiver 8000 may display, as the synthetic image, the normal captured image in which the signal transmission part is indicated by a dotted frame and an identifier (e.g. ID: 101, ID: 102, etc.), as illustrated in (c) in FIG. 13. As another alternative, the receiver 8000 may superimpose, instead of the bright line pattern, a signal identification object which is an image having a predetermined color for notifying transmission of a specific type of signal on the normal captured image to generate the synthetic image, and display the synthetic image, as illustrated in (d) in FIG. 13. In this case, the color of the signal identification object differs depending on the type of signal output from the transmitter. For example, a red signal identification object is superimposed in the case where the signal output from the transmitter is position information, and a green signal identification object is superimposed in the case where the signal output from the transmitter is a coupon.
FIG. 14 is a diagram illustrating an example of display operation of a receiver in this embodiment.
For example, in the case of receiving the signal by visible light communication, the receiver 8000 may output a sound for notifying the user that the transmitter has been discovered, while displaying the normal captured image. In this case, the receiver 8000 may change the type of output sound, the number of outputs, or the output time depending on the number of discovered transmitters, the type of received signal, the type of information specified by the signal, or the like.
FIG. 15 is a diagram illustrating another example of operation of a receiver in this embodiment.
For example, when the user touches the bright line pattern shown in the synthetic image, the receiver 8000 generates an information notification image based on the signal transmitted from the subject corresponding to the touched bright line pattern, and displays the information notification image. The information notification image indicates, for example, a coupon or a location of a store. The bright line pattern may be the signal specification object, the signal identification object, or the dotted frame illustrated in FIG. 13. The same applies to the below-mentioned bright line pattern.
FIG. 16 is a diagram illustrating another example of operation of a receiver in this embodiment.
For example, when the user touches the bright line pattern shown in the synthetic image, the receiver 8000 generates an information notification image based on the signal transmitted from the subject corresponding to the touched bright line pattern, and displays the information notification image. The information notification image indicates, for example, the current position of the receiver 8000 by a map or the like.
FIG. 17 is a diagram illustrating another example of operation of a receiver in this embodiment.
For example, the receiver 8000 receives signals from two street lightings which are subjects as transmitters. The receiver 8000 estimates the current position of the receiver 8000 based on these signals, in the same way as above. The receiver 8000 then displays the normal captured image, and also superimposes an information notification image (an image showing latitude, longitude, and the like) indicating the estimation result on the normal captured image. The receiver 8000 may also display an auxiliary information notification image on the normal captured image. For instance, the auxiliary information notification image prompts the user to perform an operation for calibrating the 9-axis sensor (particularly the geomagnetic sensor), i.e. an operation for drift cancellation. As a result of such an operation, the current position can be estimated with high accuracy.
When the user touches the displayed information notification image, the receiver 8000 may display the map showing the estimated position, instead of the normal captured image.
FIG. 18 is a diagram illustrating another example of operation of a receiver in this embodiment.
For example, when the user swipes on the receiver 8000 on which the synthetic image is displayed, the receiver 8000 displays the normal captured image including the dotted frame and the identifier like the normal captured image illustrated in (c) in FIG. 13, and also displays a list of information to follow the swipe operation. The list includes information specified by the signal transmitted from the part (transmitter) identified by each identifier. The swipe may be, for example, an operation of moving the user's finger from outside the display of the receiver 8000 on the right side into the display. The swipe may be an operation of moving the user's finger from the top, bottom, or left side of the display into the display.
When the user taps information included in the list, the receiver 8000 may display an information notification image (e.g. an image showing a coupon) indicating the information in more detail.
FIG. 19 is a diagram illustrating another example of operation of a receiver in this embodiment.
For example, when the user swipes on the receiver 8000 on which the synthetic image is displayed, the receiver 8000 superimposes an information notification image on the synthetic image, to follow the swipe operation. The information notification image indicates the subject distance with an arrow so as to be easily recognizable by the user. The swipe may be, for example, an operation of moving the user's finger from outside the display of the receiver 8000 on the bottom side into the display. The swipe may be an operation of moving the user's finger from the left, top, or right side of the display into the display.
FIG. 20 is a diagram illustrating another example of operation of a receiver in this embodiment.
For example, the receiver 8000 captures, as a subject, a transmitter which is a signage showing a plurality of stores, and displays the normal captured image obtained as a result. When the user taps a signage image of one store included in the subject shown in the normal captured image, the receiver 8000 generates an information notification image based on the signal transmitted from the signage of the store, and displays an information notification image 8001. The information notification image 8001 is, for example, an image showing the availability of the store and the like.
FIG. 21 is a diagram illustrating an example of operation of a receiver, a transmitter, and a server in this embodiment.
A transmitter 8012 as a television transmits a signal to a receiver 8011 by way of luminance change. The signal includes information prompting the user to buy content relating to a program being viewed. Having received the signal by visible light communication, the receiver 8011 displays an information notification image prompting the user to buy content, based on the signal. When the user performs an operation for buying the content, the receiver 8011 transmits at least one of information included in a SIM (Subscriber Identity Module) card inserted in the receiver 8011, a user ID, a terminal ID, credit card information, charging information, a password, and a transmitter ID, to a server 8013. The server 8013 manages a user ID and payment information in association with each other, for each user. The server 8013 specifies a user ID based on the information transmitted from the receiver 8011, and checks payment information associated with the user ID. By this check, the server 8013 determines whether or not to permit the user to buy the content. In the case of determining to permit the user to buy the content, the server 8013 transmits permission information to the receiver 8011. Having received the permission information, the receiver 8011 transmits the permission information to the transmitter 8012. Having received the permission information, the transmitter 8012 obtains the content via a network as an example, and reproduces the content.
The transmitter 8012 may transmit information including the ID of the transmitter 8012 to the receiver 8011, by way of luminance change. In this case, the receiver 8011 transmits the information to the server 8013. Having obtained the information, the server 8013 can determine that, for example, the television program is being viewed on the transmitter 8012, and conduct television program rating research.
The receiver 8011 may include information of an operation (e.g. voting) performed by the user in the above-mentioned information and transmit the information to the server 8013, to allow the server 8013 to reflect the information on the television program. An audience participation program can be realized in this way. Besides, in the case of receiving a post from the user, the receiver 8011 may include the post in the above-mentioned information and transmit the information to the server 8013, to allow the server 8013 to reflect the post on the television program, a network message board, or the like.
Furthermore, by the transmitter 8012 transmitting the above-mentioned information, the server 8013 can charge for television program viewing by paid broadcasting or on-demand TV. The server 8013 can also cause the receiver 8011 to display an advertisement, or the transmitter 8012 to display detailed information of the displayed television program or an URL of a site showing the detailed information. The server 8013 may also obtain the number of times the advertisement is displayed on the receiver 8011, the price of a product bought from the advertisement, or the like, and charge the advertiser according to the number of times or the price. Such price-based charging is possible even in the case where the user seeing the advertisement does not buy the product immediately. When the server 8013 obtains information indicating the manufacturer of the transmitter 8012 from the transmitter 8012 via the receiver 8011, the server 8013 may provide a service (e.g. payment for selling the product) to the manufacturer indicated by the information.
FIG. 22 is a diagram illustrating another example of operation of a receiver in this embodiment.
For example, the user points a camera of a receiver 8021 at a plurality of transmitters 8020 a to 8020 d as lightings. Here, the receiver 8021 is moved so that the transmitters 8020 a to 8020 d are sequentially captured as a subject. By performing visible light communication during the movement, the receiver 8021 receives a signal from each of the transmitters 8020 a to 8020 d. The signal includes information indicating the position of the transmitter. The receiver 8021 estimates the position of the receiver 8021 using the triangulation principle, based on the positions indicated by the signals received from the transmitters 8020 a to 8020 d, the detection result of the 9-axis sensor included in the receiver 8021, and the movement of the captured image. In this case, the drift of the 9-axis sensor (particularly the geomagnetic sensor) is canceled by moving the receiver 8021, so that the position can be estimated with higher accuracy.
FIG. 23 is a diagram illustrating another example of operation of a receiver in this embodiment.
For example, a receiver 8030 is a head-mounted display including a camera. When a start button is pressed, the receiver 8030 starts imaging in the visible light communication mode, i.e. visible light communication. In the case of receiving a signal by visible light communication, the receiver 8030 notifies the user of information corresponding to the received signal. The notification is made, for example, by outputting a sound from a speaker included in the receiver 8030, or by displaying an image. Visible light communication may be started not only when the start button is pressed, but also when the receiver 8030 receives a sound instructing the start or when the receiver 8030 receives a signal instructing the start by wireless communication. Visible light communication may also be started when the change width of the value obtained by a 9-axis sensor included in the receiver 8030 exceeds a predetermined range or when a bright line pattern, even if only slightly, appears in the normal captured image.
FIG. 24 is a diagram illustrating an example of initial setting of a receiver in this embodiment.
The receiver 8030 displays an alignment image 8031 upon initial setting. The alignment image 8031 is used to align the position pointed by the user in the image captured by the camera of the receiver 8030 and the image displayed on the receiver 8030. When the user places his or her fingertip at the position of a circle shown in the alignment image 8031, the receiver associates the position of the fingertip and the position of the circle, and performs alignment. That is, the position pointed by the user is calibrated.
FIG. 25 is a diagram illustrating another example of operation of a receiver in this embodiment.
The receiver 8030 specifies a signal transmission part by visible light communication, and displays a synthetic image 8034 in which a bright line pattern is shown in the part. The user performs an operation such as a tap or a double tap, on the bright line pattern. The receiver 8030 receives the operation, specifies the bright line pattern subjected to the operation, and displays an information notification image 8032 based on a signal transmitted from the part corresponding to the bright line pattern.
FIG. 26 is a diagram illustrating another example of operation of a receiver in this embodiment.
The receiver 8030 displays the synthetic image 8034 in the same way as above. The user performs an operation of moving his or her fingertip so as to encircle the bright line pattern in the synthetic image 8034. The receiver 8030 receives the operation, specifies the bright line pattern subjected to the operation, and displays an information notification image 8032 based on a signal transmitted from the part corresponding to the bright line pattern.
FIG. 27 is a diagram illustrating another example of operation of a receiver in this embodiment.
The receiver 8030 displays the synthetic image 8034 in the same way as above. The user performs an operation of placing his or her fingertip at the bright line pattern in the synthetic image 8034 for a predetermined time or more. The receiver 8030 receives the operation, specifies the bright line pattern subjected to the operation, and displays an information notification image 8032 based on a signal transmitted from the part corresponding to the bright line pattern.
FIG. 28 is a diagram illustrating another example of operation of a receiver in this embodiment.
The receiver 8030 displays the synthetic image 8034 in the same way as above. The user performs an operation of moving his or her fingertip toward the bright line pattern in the synthetic image 8034 by a swipe. The receiver 8030 receives the operation, specifies the bright line pattern subjected to the operation, and displays an information notification image 8032 based on a signal transmitted from the part corresponding to the bright line pattern.
FIG. 29 is a diagram illustrating another example of operation of a receiver in this embodiment.
The receiver 8030 displays the synthetic image 8034 in the same way as above. The user performs an operation of continuously directing his or her gaze to the bright line pattern in the synthetic image 8034 for a predetermined time or more. Alternatively, the user performs an operation of blinking a predetermined number of times while directing his or her gaze to the bright line pattern. The receiver 8030 receives the operation, specifies the bright line pattern subjected to the operation, and displays an information notification image 8032 based on a signal transmitted from the part corresponding to the bright line pattern.
FIG. 30 is a diagram illustrating another example of operation of a receiver in this embodiment.
The receiver 8030 displays the synthetic image 8034 in the same way as above, and also displays an arrow associated with each bright line pattern in the synthetic image 8034. The arrow of each bright line pattern differs in direction. The user performs an operation of moving his or her head along one of the arrows. The receiver 8030 receives the operation based on the detection result of the 9-axis sensor, and specifies the bright line pattern associated with the arrow corresponding to the operation, i.e. the arrow in the direction in which the head is moved. The receiver 8030 displays the information notification image 8032 based on the signal transmitted from the part corresponding to the bright line pattern.
FIG. 31A is a diagram illustrating a pen used to operate a receiver in this embodiment.
A pen 8033 includes a transmitter 8033 a for transmitting a signal by way of luminance change, and buttons 8033 b and 8033 c. When the button 8033 b is pressed, the transmitter 8033 a transmits a predetermined first signal. When the button 8033 c is pressed, the transmitter 8033 a transmits a predetermined second signal different from the first signal.
FIG. 31B is a diagram illustrating operation of a receiver using a pen in this embodiment.
The pen 8033 is used instead of the user's finger mentioned above, like a stylus pen. By selective use of the buttons 8033 b and 8033 c, the pen 8033 can be used like a normal pen or an eraser.
FIG. 32 is a diagram illustrating an example of appearance of a receiver in this embodiment.
The receiver 8030 includes a first touch sensor 8030 a and a second touch sensor 8030 b. These touch sensors are attached to the frame of the receiver 8030. For example, when the user places his or her fingertip on the first touch sensor 8030 a and moves the fingertip, the receiver 8030 moves the pointer in the image displayed to the user, according to the movement of the fingertip. When the user touches the second touch sensor 8030 b, the receiver 8030 selects the object pointed by the pointer in the image displayed to the user.
FIG. 33 is a diagram illustrating another example of appearance of a receiver in this embodiment.
The receiver 8030 includes a touch sensor 8030 c. The touch sensor 8030 c is attached to the frame of the receiver 8030. For example, when the user places his or her fingertip on the touch sensor 8030 c and moves the fingertip, the receiver 8030 moves the pointer in the image displayed to the user, according to the movement of the fingertip. When the user presses the touch sensor 8030 c, the receiver 8030 selects the object pointed by the pointer in the image displayed to the user. The touch sensor 8030 c is thus realized as a clickable touch sensor.
FIG. 34 is a diagram illustrating another example of operation of a receiver in this embodiment.
The receiver 8030 displays the synthetic image 8034 in the same way as above, and also displays a pointer 8035 in the synthetic image 8034. In the case where the receiver 8030 includes the first touch sensor 8030 a and the second touch sensor 8030 b, the user places his or her fingertip on the first touch sensor 8030 a and moves the fingertip, to move the pointer to the object as the bright line pattern. The user then touches the second touch sensor 8030 b, to cause the receiver 8030 to select the bright line pattern. Having selected the bright line pattern, the receiver 8030 displays the information notification image 8032 based on the signal transmitted from the part corresponding to the bright line pattern.
In the case where the receiver 8030 includes the touch sensor 8030 c, the user places his or her fingertip on the touch sensor 8030 c and moves the fingertip, to move the pointer to the object as the bright line pattern. The user then presses the touch sensor 8030 c, to cause the receiver 8030 to select the bright line pattern. Having selected the bright line pattern, the receiver 8030 displays the information notification image 8032 based on the signal transmitted from the part corresponding to the bright line pattern.
FIG. 35A is a diagram illustrating another example of operation of a receiver in this embodiment.
The receiver 8030 displays a gesture confirmation image 8036 based on a signal obtained by visible light communication. The gesture confirmation image 8036 prompts the user to make a predetermined gesture, to provide a service to the user as an example.
FIG. 35B is a diagram illustrating an example of application using a receiver in this embodiment.
A user 8038 carrying the receiver 8030 is in a shop or the like. Here, the receiver 8030 displays the above-mentioned gesture confirmation image 8036 to the user 8038. The user 8038 makes the predetermined gesture according to the gesture confirmation image 8036. A staff 8039 in the shop carries a receiver 8037. The receiver 8037 is a head-mounted display including a camera, and may have the same structure as the receiver 8030. The receiver 8037 displays the gesture confirmation image 8036 based on a signal obtained by visible light communication, too. The staff 8039 determines whether or not the predetermined gesture indicated by the displayed gesture confirmation image 8036 and the gesture made by the user 8038 match. In the case of determining that the predetermined gesture and the gesture made by the user 8038 match, the staff 8039 provides the service associated with the gesture confirmation image 8036, to the user 8038.
FIG. 36A is a diagram illustrating another example of operation of a receiver in this embodiment.
The receiver 8030 displays a gesture confirmation image 8040 based on a signal obtained by visible light communication. The gesture confirmation image 8040 prompts the user to make a predetermined gesture, to permit wireless communication as an example.
FIG. 36B is a diagram illustrating an example of application using a receiver in this embodiment.
The user 8038 carries the receiver 8030. Here, the receiver 8030 displays the above-mentioned gesture confirmation image 8040 to the user 8038. The user 8038 makes the predetermined gesture according to the gesture confirmation image 8040. A person around the user 8038 carries the receiver 8037. The receiver 8037 is a head-mounted display including a camera, and may have the same structure as the receiver 8030. The receiver 8037 captures the predetermined gesture made by the user 8038, to obtain authentication information such as a password included in the gesture. In the case where the receiver 8037 determines that the authentication information matches predetermined information, the receiver 8037 establishes wireless connection with the receiver 8030. Subsequently, the receivers 8030 and 8037 can wirelessly communicate with each other.
FIG. 37A is a diagram illustrating an example of operation of a transmitter in this embodiment.
The transmitter alternately transmits signals 1 and 2, for example in a predetermined period. The transmission of the signal 1 and the transmission of the signal 2 are each carried out by way of luminance change such as blinking of visible light. A luminance change pattern for transmitting the signal 1 and a luminance change pattern for transmitting the signal 2 are different from each other.
FIG. 37B is a diagram illustrating another example of operation of a transmitter in this embodiment.
The transmitter may transmit the signals 1 and 2 intermittently with a buffer time, instead of continuously transmitting the signals 1 and 2 as mentioned above. In the buffer time, the transmitter does not change in luminance. Alternatively, in the buffer time, the transmitter may transmit a signal indicating that the transmitter is in the buffer time by way of luminance change, or perform a luminance change different from the luminance change for transmitting the signal 1 or the luminance change for transmitting the signal 2. This enables the receiver to appropriately receive the signals 1 and 2 without interference.
FIG. 38 is a diagram illustrating another example of operation of a transmitter in this embodiment.
The transmitter repeatedly transmits a signal sequence made up of a preamble, a block 1, a block 2, a block 3, and a check signal, by way of luminance change. The block 1 includes a preamble, an address 1, data 1, and a check signal. The blocks 2 and 3 each have the same structure as the block 1. Specific information is obtained by using data included in the blocks 1, 2, and 3.
In detail, in the above-mentioned signal sequence, one set of data or information is stored in a state of being divided into three blocks. Accordingly, even when a receiver that needs a blanking time for imaging cannot receive all data of the blocks 1, 2, and 3 from one signal sequence, the receiver can receive the remaining data from another signal sequence. As a result, even a receiver that needs a blanking time can appropriately obtain the specific information from at least one signal sequence.
In the above-mentioned signal sequence, a preamble and a check signal are provided for a set of three blocks. Hence, a receiver capable of receiving light without needing a blanking time, such as a receiver including an illuminance sensor, can receive one signal sequence at one time through the use of the preamble and the check signal provided for the set, thus obtaining the specific information in a short time.
FIG. 39 is a diagram illustrating another example of operation of a transmitter in this embodiment.
When repeatedly transmitting the signal sequence including the blocks 1, 2, and 3 as described above, the transmitter may change, for each signal sequence, the order of the blocks included in the signal sequence. For example, the blocks 1, 2, and 3 are included in this order in the first signal sequence, and the blocks 3, 1, and 2 are included in this order in the next signal sequence. A receiver that requires a periodic blanking time can therefore avoid obtaining only the same block.
FIG. 40 is a diagram illustrating an example of communication form between a plurality of transmitters and a receiver in this embodiment.
A receiver 8050 may receive signals (visible light) transmitted from transmitters 8051 a and 8051 b as lightings and reflected by a reflection surface. The receiver 8050 can thus receive signals from many transmitters all together. In this case, the transmitters 8051 a and 8051 b transmit signals of different frequencies or protocols. As a result, the receiver 8050 can receive the signals from the transmitters without interference.
FIG. 41 is a diagram illustrating an example of operation of a plurality of transmitters in this embodiment.
One of the transmitters 8051 a and 8051 b may monitor the signal transmission state of the other transmitter, and transmit a signal to avoid interference with a signal of the other transmitter. For instance, one transmitter receives a signal transmitted from the other transmitter, and transmits a signal of a protocol different from the received signal. Alternatively, one transmitter detects a time period during which no signal is transmitted from the other transmitter, and transmits a signal during the time period.
FIG. 42 is a diagram illustrating another example of communication form between a plurality of transmitters and a receiver in this embodiment.
The transmitters 8051 a and 8051 b may transmit signals of the same frequency or protocol. In this case, the receiver 8050 specifies the strength of the signal transmitted from each of the transmitters, i.e. the edge strength of the bright line included in the captured image. The strength is lower when the distance between the receiver 8050 and the transmitter is longer. In the case where the distance between the receiver 8050 and the transmitter 8051 a and the distance between the receiver 8050 and the transmitter 8051 b are different from each other, the difference in distance can be exploited in this way. Thus, the receiver 8050 can separately receive the signals transmitted from the transmitters 8051 a and 8051 b appropriately, according to the specified strengths.
FIG. 43 is a diagram illustrating another example of operation of a receiver in this embodiment.
The receiver 8050 receives a signal transmitted from the transmitter 8051 a and reflected by a reflection surface. Here, the receiver 8050 may estimate the position of the transmitter 8051 a, based on the strength distribution of luminance (the difference in luminance between a plurality of positions) in the captured image.
FIG. 44 is a diagram illustrating an example of application of a receiver in this embodiment.
A receiver 7510 a such as a smartphone captures a light source 7510 b by a back camera (out camera) 7510 c to receive a signal transmitted from the light source 7510 b, and obtains the position and direction of the light source 7510 b from the received signal. The receiver 7510 a estimates the position and direction of the receiver 7510 a, from the state of the light source 7510 b in the captured image and the sensor value of the 9-axis sensor included in the receiver 7510 a. The receiver 7510 a captures a user 7510 e by a front camera (face camera, in camera) 7510 f, and estimates the position and direction of the head and the gaze direction (the position and direction of the eye) of the user 7510 e by image processing. The receiver 7510 a transmits the estimation result to the server. The receiver 7510 a changes the behavior (display content or playback sound) according to the gaze direction of the user 7510 e. The imaging by the back camera 7510 c and the imaging by the front camera 7510 f may be performed simultaneously or alternately.
FIG. 45 is a diagram illustrating an example of application of a receiver in this embodiment.
Receivers 7511 d and 7511 i such as smartphones respectively receive signals from light sources 7511 b and 7511 g, estimate the positions and directions of the receivers 7511 d and 7511 i, and estimate the gaze directions of users 7511 e and 7511 i, as in the above-mentioned way. The receivers 7511 d and 7511 i respectively obtain information of surrounding objects 7511 a to 7511 c and 7511 f to 7511 h from a server, based on the received data. The receivers 7511 d and 7511 i change their display contents as if the users can see the objects on the opposite side through the receivers 7511 d and 7511 i. The receivers 7511 d and 7511 i display an AR (Augmented Reality) object such as 7511 k, according to the display contents. When the gaze of the user 7511 j exceeds the imaging range of the camera, the receiver 7511 i displays that the range is exceeded, as in 7511 l. As an alternative, the receiver 7511 i displays an AR object or other information in the area outside the range. As another alternative, the receiver 7511 i displays a previously captured image in the area outside the range in a state of being connected to the current image.
FIG. 46 is a diagram illustrating an example of application of a receiver in this embodiment.
A receiver 7512 c such as a smartphone receives a signal from a light source 7512 a, estimates the position and direction of the receiver 7512 c, and estimates the gaze direction of a user 7512 d, as in the above-mentioned way. The receiver 7512 c performs a process relating to an object 7512 b in the gaze direction of the user 7512 d. For example, the receiver 7512 c displays information about the object 7512 b on the screen. When the gaze direction of a user 7512 h moves from an object 7512 f to a receiver 7512 g, the receiver 7512 g determines that the user 7512 h is interested in the object 7512 h, and continues the process relating to the object 7512 h. For example, the receiver 7512 g keeps displaying the information of the object 7512 f on the screen.
FIG. 47 is a diagram illustrating an example of application of a transmitter in this embodiment.
A transmitter 7513 a such as a lighting is high in luminance. Regardless of whether the luminance is high or low as a transmission signal, the transmitter 7513 a captured by a receiver exceeds an upper limit of brightness, and as a result no bright line appears as in 7513 b. Accordingly, a transmitter 7513 c includes a part 7513 d such as a diffusion plate or a prism for diffusing or weakening light, to reduce the luminance. As a result, the receiver can capture bright lines as in 7513 e.
FIG. 48 is a diagram illustrating an example of application of a transmitter in this embodiment.
A transmitter 7514 a such as a lighting does not have a uniform light source, and so the luminance is not uniform in a captured image 7514 b, causing a reception error. Accordingly, a transmitter 7514 c includes a part 7514 d such as a diffusion plate or a prism for diffusing light, to attain uniform luminance as in 7514 c. A reception error can be prevented in this way.
FIG. 49 is a diagram illustrating an example of application of a receiver in this embodiment.
Transmitters 7515 a and 7515 b are each high in luminance in the center part, so that bright lines appear not in the center part but in the peripheral part in an image captured by a receiver. Since the bright lines are discontinuous, the receiver cannot receive a signal from a part 7515 d, but can receive a signal from a part 7515 c. By reading bright lines along a path 7515 e, the receiver can receive a signal from more bright lines than in the part 7515 c.
FIG. 50 is a diagram illustrating an example of application of a transmitter in this embodiment.
Transmitters 7516 a, 7516 b, 7516 c, and 7516 d such as lightings are high in luminance like 7513 a, and bright lines tend not to appear when captured by a receiver. Accordingly, a diffusion plate/prism 7516 e, a reflection plate 7516 f, a reflection plate/half mirror 7516 g, a reflection plate 7516 h, or a diffusion plate/prism 7516 j is included to diffuse light, with it being possible to widen the part where bright lines appear. These transmitters are each captured with bright lines appearing in the periphery, like 7515 a. Since the receiver estimates the distance between the receiver and the transmitter using the size of the transmitter in the captured image, the part where light is diffused is set as the size of the light source and stored in a server or the like in association with the transmission ID, as a result of which the receiver can accurately estimate the distance to the transmitter.
FIG. 51 is a diagram illustrating an example of application of a transmitter in this embodiment.
A transmitter 7517 a such as a lighting is high in luminance like 7513 a, and bright lines tend not to appear when captured by a receiver. Accordingly, a reflection plate 7517 b is included to diffuse light, with it being possible to widen the part where bright lines appear.
FIG. 52 is a diagram illustrating an example of application of a transmitter in this embodiment.
A transmitter 7518 a reflects light from a light source by a reflection plate 7518 c, as a result of which a receiver can capture bright lines in a wide range. A transmitter 7518 d directs a light source toward a diffusion plate or prism 7518 e, as a result of which a receiver can capture bright lines in a wide range.
FIG. 53 is a diagram illustrating another example of operation of a receiver in this embodiment.
A receiver displays a bright line pattern using the above-mentioned synthetic image, intermediate image, or the like. Here, the receiver may be incapable of receiving a signal from a transmitter corresponding to the bright line pattern. When the user performs an operation (e.g. a tap) on the bright line pattern to select the bright line pattern, the receiver displays the synthetic image or intermediate image in which the bright line pattern is enlarged by optical zoom. Through such optical zoom, the receiver can appropriately receive the signal from the transmitter corresponding to the bright line pattern. That is, even when the captured image is too small to obtain the signal, the signal can be appropriately received by performing optical zoom. In the case where the displayed image is large enough to obtain the signal, too, faster reception is possible by optical zoom.
(Summary of this Embodiment)
An information communication method in this embodiment is an information communication method of obtaining information from a subject, the information communication method including: setting an exposure time of an image sensor so that, in an image obtained by capturing the subject by the image sensor, a bright line corresponding to an exposure line included in the image sensor appears according to a change in luminance of the subject; obtaining a bright line image by capturing the subject that changes in luminance by the image sensor with the set exposure time, the bright line image being an image including the bright line; displaying, based on the bright line image, a display image in which the subject and surroundings of the subject are shown, in a form that enables identification of a spatial position of a part where the bright line appears; and obtaining transmission information by demodulating data specified by a pattern of the bright line included in the obtained bright line image.
In this way, a synthetic image or an intermediate image illustrated in, for instance, FIGS. 7 to 9 and 13 is displayed as the display image. In the display image in which the subject and the surroundings of the subject are shown, the spatial position of the part where the bright line appears is identified by a bright line pattern, a signal specification object, a signal identification object, a dotted frame, or the like. By looking at such a display image, the user can easily find the subject that is transmitting the signal through the change in luminance.
For example, the information communication method may further include: setting a longer exposure time than the exposure time; obtaining a normal captured image by capturing the subject and the surroundings of the subject by the image sensor with the longer exposure time; and generating a synthetic image by specifying, based on the bright line image, the part where the bright line appears in the normal captured image, and superimposing a signal object on the normal captured image, the signal object being an image indicating the part, wherein in the displaying, the synthetic image is displayed as the display image.
In this way, the signal object is, for example, a bright line pattern, a signal specification object, a signal identification object, a dotted frame, or the like, and the synthetic image is displayed as the display image as illustrated in FIGS. 8, 9, and 13. Hence, the user can more easily find the subject that is transmitting the signal through the change in luminance.
For example, in the setting of an exposure time, the exposure time may be set to 1/3000 second, in the obtaining of a bright line image, the bright line image in which the surroundings of the subject are shown may be obtained, and in the displaying, the bright line image may be displayed as the display image.
In this way, the bright line image is obtained and displayed as an intermediate image, for instance as illustrated in FIG. 7. This eliminates the need for a process of obtaining a normal captured image and a visible light communication image and synthesizing them, thus contributing to a simpler process.
For example, the image sensor may include a first image sensor and a second image sensor, in the obtaining of the normal captured image, the normal captured image may be obtained by image capture by the first image sensor, and in the obtaining of a bright line image, the bright line image may be obtained by image capture by the second image sensor simultaneously with the first image sensor.
In this way, the normal captured image and the visible light communication image which is the bright line image are obtained by the respective cameras, for instance as illustrated in FIG. 9. As compared with the case of obtaining the normal captured image and the visible light communication image by one camera, the images can be obtained promptly, contributing to a faster process.
For example, the information communication method may further include presenting, in the case where the part where the bright line appears is designated in the display image by an operation by a user, presentation information based on the transmission information obtained from the pattern of the bright line in the designated part. Examples of the operation by the user include: a tap; a swipe; an operation of continuously placing the user's fingertip on the part for a predetermined time or more; an operation of continuously directing the user's gaze to the part for a predetermined time or more; an operation of moving a part of the user's body according to an arrow displayed in association with the part; an operation of placing a pen tip that changes in luminance on the part; and an operation of pointing to the part with a pointer displayed in the display image by touching a touch sensor.
In this way, the presentation information is displayed as an information notification image, for instance as illustrated in FIGS. 15 to 20 and 25 to 34. Desired information can thus be presented to the user.
For example, the image sensor may be included in a head-mounted display, and in the displaying, the display image may be displayed by a projector included in the head-mounted display.
In this way, the information can be easily presented to the user, for instance as illustrated in FIGS. 23 to 30.
For example, an information communication method of obtaining information from a subject may include: setting an exposure time of an image sensor so that, in an image obtained by capturing the subject by the image sensor, a bright line corresponding to an exposure line included in the image sensor appears according to a change in luminance of the subject; obtaining a bright line image by capturing the subject that changes in luminance by the image sensor with the set exposure time, the bright line image being an image including the bright line; and obtaining the information by demodulating data specified by a pattern of the bright line included in the obtained bright line image, wherein in the obtaining of a bright line image, the bright line image including a plurality of parts where the bright line appears is obtained by capturing a plurality of subjects in a period during which the image sensor is being moved, and in the obtaining of the information, a position of each of the plurality of subjects is obtained by demodulating, for each of the plurality of parts, the data specified by the pattern of the bright line in the part, and the information communication method may further include estimating a position of the image sensor, based on the obtained position of each of the plurality of subjects and a moving state of the image sensor.
In this way, the position of the receiver including the image sensor can be accurately estimated based on the changes in luminance of the plurality of subjects such as lightings, for instance as illustrated in FIG. 22.
For example, an information communication method of obtaining information from a subject may include: setting an exposure time of an image sensor so that, in an image obtained by capturing the subject by the image sensor, a bright line corresponding to an exposure line included in the image sensor appears according to a change in luminance of the subject; obtaining a bright line image by capturing the subject that changes in luminance by the image sensor with the set exposure time, the bright line image being an image including the bright line; obtaining the information by demodulating data specified by a pattern of the bright line included in the obtained bright line image; and presenting the obtained information, wherein in the presenting, an image prompting to make a predetermined gesture is presented to a user of the image sensor as the information.
In this way, user authentication and the like can be conducted according to whether or not the user makes the gesture as prompted, for instance as illustrated in FIGS. 35A to 36B. This enhances convenience.
For example, an information communication method of obtaining information from a subject may include: setting an exposure time of an image sensor so that, in an image obtained by capturing the subject by the image sensor, a bright line corresponding to an exposure line included in the image sensor appears according to a change in luminance of the subject; obtaining a bright line image by capturing the subject that changes in luminance by the image sensor with the set exposure time, the bright line image being an image including the bright line; and obtaining the information by demodulating data specified by a pattern of the bright line included in the obtained bright line image, wherein in the obtaining of a bright line image, the bright line image is obtained by capturing a plurality of subjects reflected on a reflection surface, and in the obtaining of the information, the information is obtained by separating a bright line corresponding to each of the plurality of subjects from bright lines included in the bright line image according to a strength of the bright line and demodulating, for each of the plurality of subjects, the data specified by the pattern of the bright line corresponding to the subject.
In this way, even in the case where the plurality of subjects such as lightings each change in luminance, appropriate information can be obtained from each subject, for instance as illustrated in FIG. 42.
For example, an information communication method of obtaining information from a subject may include: setting an exposure time of an image sensor so that, in an image obtained by capturing the subject by the image sensor, a bright line corresponding to an exposure line included in the image sensor appears according to a change in luminance of the subject; obtaining a bright line image by capturing the subject that changes in luminance by the image sensor with the set exposure time, the bright line image being an image including the bright line; and obtaining the information by demodulating data specified by a pattern of the bright line included in the obtained bright line image, wherein in the obtaining of a bright line image, the bright line image is obtained by capturing the subject reflected on a reflection surface, and the information communication method may further include estimating a position of the subject based on a luminance distribution in the bright line image.
In this way, the appropriate position of the subject can be estimated based on the luminance distribution, for instance as illustrated in FIG. 43.
For example, an information communication method of transmitting a signal using a change in luminance may include: determining a first pattern of the change in luminance, by modulating a first signal to be transmitted; determining a second pattern of the change in luminance, by modulating a second signal to be transmitted; and transmitting the first signal and the second signal by a light emitter alternately changing in luminance according to the determined first pattern and changing in luminance according to the determined second pattern.
In this way, the first signal and the second signal can each be transmitted without a delay, for instance as illustrated in FIG. 37A.
For example, in the transmitting, a buffer time may be provided when switching the change in luminance between the change in luminance according to the first pattern and the change in luminance according to the second pattern.
In this way, interference between the first signal and the second signal can be suppressed, for instance as illustrated in FIG. 37B.
For example, an information communication method of transmitting a signal using a change in luminance may include: determining a pattern of the change in luminance by modulating the signal to be transmitted; and transmitting the signal by a light emitter changing in luminance according to the determined pattern, wherein the signal is made up of a plurality of main blocks, each of the plurality of main blocks includes first data, a preamble for the first data, and a check signal for the first data, the first data is made up of a plurality of sub-blocks, and each of the plurality of sub-blocks includes second data, a preamble for the second data, and a check signal for the second data.
In this way, data can be appropriately obtained regardless of whether or not the receiver needs a blanking time, for instance as illustrated in FIG. 38.
For example, an information communication method of transmitting a signal using a change in luminance may include: determining, by each of a plurality of transmitters, a pattern of the change in luminance by modulating the signal to be transmitted; and transmitting, by each of the plurality of transmitters, the signal by a light emitter in the transmitter changing in luminance according to the determined pattern, wherein in the transmitting, the signal of a different frequency or protocol is transmitted.
In this way, interference between signals from the plurality of transmitters can be suppressed, for instance as illustrated in FIG. 40.
For example, an information communication method of transmitting a signal using a change in luminance may include: determining, by each of a plurality of transmitters, a pattern of the change in luminance by modulating the signal to be transmitted; and transmitting, by each of the plurality of transmitters, the signal by a light emitter in the transmitter changing in luminance according to the determined pattern, wherein in the transmitting, one of the plurality of transmitters receives a signal transmitted from a remaining one of the plurality of transmitters, and transmits an other signal in a form that does not interfere with the received signal.
In this way, interference between signals from the plurality of transmitters can be suppressed, for instance as illustrated in FIG. 41.
This embodiment describes each example of application using a receiver such as a smartphone and a transmitter for transmitting information as a blink pattern of an LED, an organic EL device, or the like in Embodiment 1 or 2 described above.
First, a receiver receives a signal by an illuminance sensor (Step 8101). Next, the receiver obtains information such as position information from a server, based on the received signal (Step 8102). The receiver then activates an image sensor capable of capturing the light reception direction of the illuminance sensor (Step 8103). The receiver receives all or part of a signal by the image sensor, and determines whether or not all or part of the signal is the same as the signal received by the illuminance sensor (Step 8104). Following this, the receiver estimates the position of the receiver, from the position of the transmitter in the captured image, information from a 9-axis sensor included in the receiver, and the position information of the transmitter (Step 8105). Thus, the receiver activates the illuminance sensor of low power consumption and, in the case where the signal is received by the illuminance sensor, activates the image sensor. The receiver then performs position estimation using image capture by the image sensor. In this way, the position of the receiver can be accurately estimated while saving power.
A receiver recognizes a periodic change of luminance from the sensor value of an illuminance sensor (Step 8111). The receiver then activates an image sensor capable of capturing the light reception direction of the illuminance sensor, and receives a signal (Step 8112). Thus, the receiver activates the illuminance sensor of low power consumption and, in the case where the periodic change of luminance is received by the illuminance sensor, activates the image sensor, in the same way as above. The receiver then receives the accurate signal using image capture by the image sensor. In this way, the accurate signal can be received while saving power.
FIG. 56A is a diagram illustrating an example of operation of a transmitter in Embodiment 3.
A transmitter 8115 includes a power supply unit 8115 a, a signal control unit 8115 b, a light emitting unit 8115 c, and a light emitting unit 8115 d. The power supply unit 8115 a supplies power to the signal control unit 8115 b. The signal control unit 8115 b divides the power supplied from the power supply unit 8115 a into the light emitting units 8115 c and 8115 d, and controls the luminance changes of the light emitting units 8115 c and 8115 d.
FIG. 56B is a diagram illustrating another example of operation of a transmitter in Embodiment 3.
A transmitter 8116 includes a power supply unit 8116 a, a signal control unit 8116 b, a light emitting unit 8116 c, and a light emitting unit 8116 d. The power supply unit 8116 a supplies power to the light emitting units 8116 c and 8116 d. The signal control unit 8116 b controls the power supplied from the power supply unit 8116 a, thereby controlling the luminance changes of the light emitting units 8116 c and 8116 d. The power use efficiency can be enhanced by the signal control unit 8116 b controlling the power supply unit 8116 a that supplies power to each of the light emitting units 8116 c and 8116 d.
The system includes a centralized control unit 8118, a transmitter 8117, and a transmitter 8120. The centralized control unit 8118 controls signal transmission by a change in luminance of each of the transmitters 8117 and 8120. For example, the centralized control unit 8118 causes the transmitters 8117 and 8120 to transmit the same signal at the same time, or causes one of the transmitters to transmit a signal unique to the transmitter.
The transmitter 8120 includes two transmission units 8121 and 8122, a signal change unit 8123, a signal storage unit 8124, a synchronous signal input unit 8125, a synchronous control unit 8126, and a light receiving unit 8127.
The two transmission units 8121 and 8122 each have the same structure as the transmitter 8115 illustrated in FIG. 56A, and transmits a signal by changing in luminance. In detail, the transmission unit 8121 includes a power supply unit 8121 a, a signal control unit 8121 b, a light emitting unit 8121 c, and a light emitting unit 8121 d. The transmission unit 8122 includes a power supply unit 8122 a, a signal control unit 8122 b, a light emitting unit 8122 c, and a light emitting unit 8122 d.
The signal change unit 8123 modulates a signal to be transmitted, to a signal indicating a luminance change pattern. The signal storage unit 8124 stores the signal indicating the luminance change pattern. The signal control unit 8121 b in the transmission unit 121 reads the signal stored in the signal storage unit 8124, and causes the light emitting units 8121 c and 8121 d to change in luminance according to the signal.
The synchronous signal input unit 8125 obtains a synchronous signal according to control by the centralized control unit 8118. The synchronous control unit 8126 synchronizes the luminance changes of the transmission units 8121 and 8122, when the synchronous signal is obtained. That is, the synchronous control unit 8126 controls the signal control units 8121 b and 8122 b, to synchronize the luminance changes of the transmission units 8121 and 8122. Here, the light receiving unit 8127 detects light emission from the transmission units 8121 and 8122. The synchronous control unit 8126 feedback-controls the signal control units 8121 b and 8122 b, according to the light detected by the light receiving unit 8127.
A transmitter 8130 includes a transmission unit 8131 that transmits a signal by changing in luminance, and a non-transmission unit 8132 that emits light without transmitting a signal.
The transmission unit 8131 has the same structure as the transmitter 8115 illustrated in FIG. 56A, and includes a power supply unit 8131 a, a signal control unit 8131 b, and light emitting units 8131 c to 8131 f. The non-transmission unit 8132 includes a power supply unit 8132 a and light emitting units 8132 c to 8132 f, but does not include a signal control unit. In other words, in the case where there are a plurality of units each including a power supply and luminance change synchronous control cannot be performed between the plurality of units, a signal control unit is provided in only one of the plurality of units to cause the unit to change in luminance, as in the structure illustrated in FIG. 58.
In the transmitter 8130, the light emitting units 8131 c to 8131 f in the transmission unit 8131 are continuously arranged in a line. That is, none of the light emitting units 8132 c to 8132 f in the non-transmission unit 8132 is mixed in the set of the light emitting units 8131 c to 8131 f. This makes the light emitter that changes in luminance larger in size, so that the receiver can easily receive the signal transmitted using the change in luminance.
A transmitter 8134 such as a signage includes three light emitting units (light emitting areas) 8134 a to 8134 c. Light from these light emitting units 8134 a to 8134 c do not interfere with each other. In the case where only one of the light emitting units 8134 a to 8134 c can be changed in luminance to transmit a signal, it is desirable to change in luminance the light emitting unit 8134 b at the center, as illustrated in (a) in FIG. 59A. In the case where two of the light emitting units 8134 a to 8134 c can be changed in luminance, it is desirable to change in luminance the light emitting unit 8134 b at the center and the light emitting unit 8134 a or 8134 c at either edge, as illustrated in (b) in FIG. 59A. Changing in luminance the light emitting units at such positions enables the receiver to appropriately receive the signal transmitted using the change in luminance.
A transmitter 8135 such as a signage includes three light emitting units 8135 a to 8135 c. Light from adjacent light emitting units of these light emitting units 8135 a to 8135 c interferes with each other. In the case where only one of the light emitting units 8135 a to 8135 c can be changed in luminance to transmit a signal, it is desirable to change in luminance the light emitting unit 8135 a or 8135 c at either edge, as illustrated in (a) in FIG. 59B. This prevents light from another light emitting unit from interfering with the luminance change for signal transmission. In the case where two of the light emitting units 8135 a to 8135 c can be changed in luminance, it is desirable to change in luminance the light emitting unit 8135 b at the center and the light emitting unit 8135 a or 8135 c at either edge, as illustrated in (b) in FIG. 59B. Changing in luminance the light emitting units at such positions contributes to a larger luminance change area, and so enables the receiver to appropriately receive the signal transmitted using the change in luminance.
In the case where two of the light emitting units 8134 a to 8134 c can be changed in luminance in the transmitter 8134, the light emitting units 8134 a and 8134 c at both edges may be changed in luminance, as illustrated in FIG. 50C. In this case, the imaging range in which the luminance change part is shown can be widened in the image capture by the receiver.
A transmitter 8137 such as a signage transmits a signal by a character part “A Shop” and a light emitting unit 8137 a changing in luminance. For example, the light emitting unit 8137 a is formed like a horizontally long rectangle, and uniformly changes in luminance. The uniform change in luminance of the light emitting unit 8137 a enables the receiver to appropriately receive the signal transmitted using the change in luminance.
A transmitter 8138 such as a signage transmits a signal by a character part “A Shop” and a light emitting unit 8138 a changing in luminance. For example, the light emitting unit 8138 a is formed like a frame along the edges of the signage, and uniformly changes in luminance. That is, the light emitting unit 8138 a is formed so that, when the light emitting unit is projected onto an arbitrary straight line, the length of the continuous projection part is at the maximum. The uniform change in luminance of the light emitting unit 8138 a enables the receiver to more appropriately receive the signal transmitted using the change in luminance.
A receiver 8142 such as a smartphone obtains position information indicating the position of the receiver 8142, and transmits the position information to a server 8141. For example, the receiver 8142 obtains the position information when using a GPS or the like or receiving another signal. The server 8141 transmits an ID list associated with the position indicated by the position information, to the receiver 8142. The ID list includes each ID such as “abcd” and info