Patent ID: 12252171

DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of an information terminal, a control method, and a control program according to the present invention will be described with reference to the accompanying drawings. The drawings are viewed in directions of reference numerals. In addition, in the present specification and the like, in order to simplify and clarify the description, a front-rear direction, a left-right direction, and an upper-lower direction are described according to directions viewed from a driver of a vehicle10shown inFIGS.1and2. In the drawings, a front side of the vehicle10is denoted by Fr, a rear side thereof is denoted by Rr, a left side thereof is denoted by L, a right side thereof is denoted by R, an upper side thereof is denoted by U, and a lower side thereof is denoted by D.

<Vehicle10Whose Movement is Controlled by Information Terminal of Present Invention>

FIG.1is a side view of the vehicle10whose movement is controlled by the information terminal of the present invention.FIG.2is atop view of the vehicle10shown inFIG.1. The vehicle10is an example of a moving object in the present invention.

The vehicle10is an automobile including a drive source (not shown) and wheels including driving wheels driven by power of the drive source and steering wheels that are steerable. In the present embodiment, the vehicle10is a four-wheeled automobile including a pair of left and right front wheels and a pair of left and right rear wheels. The drive source of the vehicle10is, for example, an electric motor. The drive source of the vehicle10may also be an internal combustion engine such as a gasoline engine or a diesel engine, or a combination of an electric motor and an internal combustion engine. In addition, the drive source of the vehicle10may drive the pair of left and right front w % heels, may drive the pair of left and right rear wheels, or may drive four wheels, that is, the pair of left and right front wheels and the pair of left and right rear wheels. The front wheels and the rear wheels may both be steering wheels that are steerable, or the front wheels or the rear wheels may be steering wheels that are steerable.

The vehicle10further includes side mirrors11L and11R. The side mirrors11L and11R are mirrors (rearview mirrors) that are provided outside front seat doors of the vehicle10for the driver to check the rear side and a rear lateral side. Each of the side mirrors11L and11R is fixed to a body of the vehicle10by a rotation shaft extending in a vertical direction, and can be opened and closed by rotating about the rotation shaft.

The vehicle10further includes a front camera12Fr, a rear camera12Rr, a left side camera12L, and a right side camera12R. The front camera12Fr is a digital camera that is provided at a front portion of the vehicle10and captures an image of the front side of the vehicle10. The rear camera12Rr is a digital camera that is provided at a rear portion of the vehicle10and captures an image of the rear side of the vehicle10. The left side camera12L is a digital camera that is provided on the left side mirror11L of the vehicle10and captures an image of the left side of the vehicle10. The right side camera12R is a digital camera that is provided on the right side mirror11R of the vehicle10and captures an image of the right side of the vehicle10.

<Internal Configuration of Vehicle10>

FIG.3is a block diagram showing an example of an internal configuration of the vehicle10shown inFIG.1. As shown inFIG.3, the vehicle10includes a sensor group16, a navigation device18, a control electronic control unit (ECU)20, an electric power steering (EPS) system22, and a communication unit24. The vehicle10further includes a driving force control system26and a braking force control system28.

The sensor group16acquires various detection values used for control performed by the control ECU20. The sensor group16includes the front camera12Fr, the rear camera12Rr, the left side camera12L, and the right side camera12R. In addition, the sensor group16includes a front sonar group32a, a rear sonar group32b, a left side sonar group32c, and a right side sonar group32d. In addition, the sensor group16includes wheel sensors34aand34b, a vehicle speed sensor36, and an operation detection unit38.

The front camera12Fr, the rear camera12Rr, the left side camera12L, and the right side camera12R acquire recognition data (for example, surrounding image) for recognizing an outside of the vehicle10by capturing images of surroundings of the vehicle10. Surrounding images captured by the front camera12Fr, the rear camera12Rr, the left side camera12L, and the right side camera12R are referred to as a front image, a rear image, a left side image, and a right side image, respectively. An image formed by the left side image and the right side image may be referred to as a side image.

The front sonar group32a, the rear sonar group32b, the left side sonar group32c, and the right side sonar group32demit sound waves to the surroundings of the vehicle10and receive reflected sounds from other objects. The front sonar group32aincludes, for example, four sonars. The sonars constituting the front sonar group32aare respectively provided on an obliquely left front side, a front left side, a front right side, and an obliquely right front side of the vehicle10. The rear sonar group32bincludes, for example, four sonars. The sonars constituting the rear sonar group32bare respectively provided on an obliquely left rear side, a rear left side, a rear right side, and an obliquely right rear side of the vehicle10. The left side sonar group32cincludes, for example, two sonars. The sonars constituting the left side sonar group32care provided in a front side and a rear side of a left side portion of the vehicle10, respectively. The right side sonar group32dincludes, for example, two sonars. The sonars constituting the right side sonar group32dare provided in a front side and in a rear side of a right side portion of the vehicle10, respectively.

The wheel sensors34aand34bdetect rotation angles of the wheels of the vehicle10. The wheel sensors34aand34bmay be implemented by angle sensors or displacement sensors. The wheel sensors34aand34boutput detection pulses each time the wheels rotate by predetermined angles. The detection pulses output from the wheel sensors34aand34bare used to calculate the rotation angles of the wheels and rotation speeds of the wheels. A movement distance of the vehicle10is calculated based on the rotation angles of the wheels. The wheel sensor34adetects, for example, a rotation angle θa of the left rear wheel. The wheel sensor34bdetects, for example, a rotation angle θb of the right rear wheel.

The vehicle speed sensor36detects a speed of a vehicle body of the vehicle10, that is, a vehicle speed V. and outputs the detected vehicle speed V to the control ECU20. The vehicle speed sensor36detects the vehicle speed V based on, for example, rotation of a countershaft of a transmission.

The operation detection unit38detects a content of an operation performed by a user using an operation input unit14, and outputs the detected content of the operation to the control ECU20. The operation input unit14includes, for example, various user interfaces such as a side mirror switch that switches opened and closed states of the side mirrors11L and11R, and a shift lever (a select lever or a selector).

The navigation device18detects a current position of the vehicle10by using, for example, a global positioning system (GPS), and guides the user along a route toward a destination. The navigation device18includes a storage device (not shown) that includes a map information database.

The navigation device18includes a touch panel42and a speaker44. The touch panel42functions as an input device and a display device of the control ECU20. The speaker44outputs various types of guide information to the user of the vehicle10by voice.

The touch panel42is configured to input various commands to the control ECU20. For example, the user can input a command related to movement assistance of the vehicle10via the touch panel42. The movement assistance includes parking assistance and exit assistance of the vehicle10. In addition, the touch panel42is configured to display various screens related to a control content of the control ECU20. For example, a screen related to the movement assistance of the vehicle10is displayed on the touch panel42. Specifically, a parking assistance button for requesting the parking assistance of the vehicle10and an exit assistance button for requesting the exit assistance are displayed on the touch panel42. The parking assistance button includes an autonomous parking button for requesting parking by autonomous steering of the control ECU20and a guidance parking button for requesting guidance when parking the vehicle by an operation of the driver. The exit assistance button includes an autonomous exit button for requesting exit by the autonomous steering of the control ECU20and a guidance exit button for requesting guidance w % ben the exit of the vehicle is performed by an operation of the driver. Constituent elements other than the touch panel42, for example, a smartphone or a tablet terminal may be used as the input device or the display device.

The control ECU20includes an input and output unit50, a calculation unit52, and a storage unit54. The calculation unit52is implemented by, for example, a central processing unit (CPU). The calculation unit52performs various types of control by controlling each unit based on a program stored in the storage unit54. In addition, the calculation unit52receives and outputs signals from and to each unit connected to the control ECU20via the input and output unit50.

The calculation unit52includes an autonomous parking control unit55configured to perform movement execution control of the vehicle10. The autonomous parking control unit55performs autonomous parking assistance and autonomous exit assistance of the vehicle10by autonomous steering in which a steering110is autonomously operated under control of the autonomous parking control unit55. In the autonomous parking assistance and the autonomous exit assistance, an accelerator pedal (not shown), a brake pedal (not shown), and the operation input unit14are autonomously operated. In addition, the autonomous parking control unit55performs guidance parking assistance and guidance exit assistance when the driver performs manual parking and manual exit of the vehicle10by operating the accelerator pedal, the brake pedal, and the operation input unit14.

For example, based on the recognition data of the outside of the vehicle10which is acquired by the front camera12Fr, the rear camera12Rr, the left side camera12L, and the right side camera12R and a predetermined parking space designated by the user, the autonomous parking control unit55controls autonomous parking of parking the vehicle10in the designated predetermined parking space and autonomous exit of causing the vehicle10to exit from the predetermined parking space.

The autonomous parking control unit55registers the predetermined parking space designated by the user in the storage unit54as a designated parking space. Based on the recognition data of the outside of the vehicle10acquired by the front camera12Fr, the rear camera12Rr, the left side camera12L, and the right side camera12R, the autonomous parking control unit55registers feature points related to the designated parking space in the storage unit54. Based on the recognition data of the outside of the vehicle10and the feature points of the designated parking space designated by the user, the autonomous parking control unit55controls the autonomous parking of parking the vehicle10in the designated parking space and autonomous exit of causing the vehicle10to exit from the designated parking space.

The EPS system22includes a steering angle sensor100, a torque sensor102, an EPS motor104, a resolver106, and an EPS ECU108. The steering angle sensor100detects a steering angle θst of the steering110. The torque sensor102detects a torque TQ applied to the steering110.

The EPS motor104applies a driving force or a reaction force to a steering column112connected to the steering110, thereby enabling assistance of an operation performed by an occupant on the steering110and enabling autonomous steering during the parking assistance. The resolver106detects a rotation angle θm of the EPS motor104. The EPS ECU108controls the entire EPS system22. The EPS ECU108includes an input and output unit (not shown), a calculation unit (not shown), and a storage unit (not shown).

The communication unit24enables wireless communication with another communication device120. Another communication device120is a base station, a communication device of other vehicle, a smartphone or a tablet terminal carried by the user of the vehicle10, or the like. The smartphone and the tablet are examples of the information terminal of the present invention.

The driving force control system26includes a driving ECU130. The driving force control system26executes driving force control of the vehicle10. The driving ECU130controls a driving force of the vehicle10by controlling an engine (not shown) or the like based on an operation performed by the user on the accelerator pedal (not shown).

The braking force control system28includes a braking ECU132. The braking force control system28executes braking force control of the vehicle10. The braking ECU132controls a braking force of the vehicle10by controlling a brake mechanism (not shown) or the like based on an operation performed by the user on the brake pedal (not shown).

<Movement Instruction Control Performed by Information Terminal>

FIG.4is a diagram showing an example of a state where a user M of the vehicle10performs movement instruction control of causing the vehicle10to autonomously exit from a parking space P by using a smartphone60carried by the user M from the outside of the vehicle10. The movement instruction control is an example of movement control of the present invention. The movement instruction control includes, for example, parking instruction control of autonomously parking the vehicle10in the parking space P and exit instruction control of causing the vehicle10to autonomously exit from the parking space P. The example shown inFIG.4shows a state where the exit instruction control of the vehicle10is performed.

An application capable of controlling movement of the vehicle10by transmitting and receiving information related to the movement instruction control of the vehicle10to and from the vehicle10is installed in the smartphone60. When a smartphone screen61(seeFIG.8and the like) implemented as a touch panel is touched by the user M, the smartphone60transmits to the vehicle10an instruction signal for instructing autonomous exit of the vehicle10by the wireless communication. For example, bluetooth low energy (BLE: registered trademark) is used as the wireless communication. The vehicle10receives the instruction signal transmitted from the smartphone60through the communication unit24. The autonomous parking control unit55of the vehicle10controls the autonomous exit of the vehicle10according to the instruction signal received through the communication unit24.

<Hardware Configuration of Smartphone60>

FIG.5is a diagram showing an example of a hardware configuration of the smartphone60. The smartphone60can be implemented by, for example, an information processing device80shown inFIG.5. The information processing device80includes a processor81, a memory82, a communication interface83, and a user interface84. The processor81, the memory82, the communication interface83, and the user interface84are connected by, for example, a bus85.

The processor81is a circuit that performs signal processing, and is, for example, a central processing unit (CPU) that controls the entire information processing device80. The processor81is an example of a control unit of the present invention. The processor81may be implemented by another digital circuit such as a field programmable gate array (FPGA) or a digital signal processor (DSP). In addition, the processor81may be implemented by combining a plurality of digital circuits.

The memory82includes, for example, a main memory and an auxiliary memory. The main memory is, for example, a random access memory (RAM). The main memory is used as a work area of the processor81.

The auxiliary memory is, for example, a nonvolatile memory such as a magnetic disk, an optical disk, or a flash memory. Various programs for causing the information processing device80to operate are stored in the auxiliary memory. The programs stored in the auxiliary memory are loaded onto the main memory and executed by the processor81.

In addition, the auxiliary memory may include a portable memory removable from the information processing device80. Examples of the portable memory include a universal serial bus (USB) flash drive, a memory card such as a secure digital (SD) memory card, and an external hard disk drive.

The communication interface83is a communication interface that performs the wireless communication with an outside of the information processing device80(for example, the communication unit24of the vehicle10). The communication interface83is controlled by the processor81.

The user interface84includes, for example, an input device that receives an operation input from the user M and an output device that outputs information to the user M. The input device can be implemented by, for example, a touch panel. The output device can be implemented by, for example, a display and a speaker. The user interface84is controlled by the processor81.

For example, the processor81performs the movement instruction control instructing the movement of the vehicle10. Specifically, the processor81performs the movement instruction control of the vehicle10based on a specific operation of the user M on the smartphone screen61of the smartphone60. The specific operation includes a continuous position instruction operation (for example, swiping operation), a rotation instruction operation in a predetermined rotation direction (for example, rotation swiping operation), and the like. The smartphone screen61is an example of a display unit of the present invention.

Further, the processor81performs control of generating a guidance image that prompts the specific operation of the user M on the smartphone screen61of the smartphone60and causing the smartphone screen61to display the generated guidance image. The guidance image includes, for example, a position guidance image that prompts a position instruction operation, a rotation guidance image that prompts the rotation instruction operation, and the like. Specifically, the processor81causes the smartphone screen61to display the guidance image when waiting for reception of the specific operation. The time of waiting for reception includes a situation where the specific operation performed by the user M is required, a situation where the user M is expected to perform the specific operation, and the like.

When the user M performs the specific operation based on the guidance image displayed on the smartphone screen61, the processor81stores position information related to the performed specific operation in the memory82. The processor81stores, for example, position information related to a specific operation for each of a clockwise rotation instruction operation in a clockwise direction and a counterclockwise rotation instruction operation in a counterclockwise direction. The position information related to the specific operation is information such that the user M performs the specific operation at which position on the smartphone screen61(for example, a right side position or a left side position of the screen). However, when the specific operation performed by the user M is a specific operation that is not based on the guidance image displayed on the smartphone screen61, for example, when the operation is completely different from the guidance image displayed on the smartphone screen61, the processor81does not store the operation information in the memory82. The processor81stores the position information related to the specific operation in the memory82when the processing of the movement instruction control of the vehicle10is completed, and does not store the position information when the processing is not completed.

In a case where the position information is stored in the memory82when the processor81waits for reception of the specific operation, the processor81displays the guidance image that prompts the specific operation based on the position information on the smartphone screen61. Specifically, in a case where the position information of the specific operation performed by the user M is stored in the memory82when the processor81waits for reception of the specific operation, the processor81causes the smartphone screen61to display the guidance image that prompts the specific operation and indicates a position based on the position information on the smartphone screen61. Specifically, in a case where the position information that indicates a range of the specific operation performed by the user M is in the memory82when the processor81waits for reception of the specific operation, the processor81causes the smartphone screen61to display the guidance image that prompts the specific operation and on which an image that indicates the range where the user M performs the specific operation on the smartphone screen61is superimposed. The position based on the position information indicates a narrow area on the smartphone screen61, and the range where the specific operation is performed indicates a relatively wide area on the smartphone screen61.

In a case where the position information is in the memory82when the processor81waits for reception of the specific operation, the processor81changes the image that indicates the range where the specific operation is performed based on the history of receiving the specific operation from the user M. Specifically, when the number of times of the reception of the same specific operation increases, gradation display of the image that indicates the range where the specific operation is performed is lightly displayed or hidden. The image that indicates the range where the previous-time specific operation is performed is displayed darker, and the image that indicates the range where the second previous-time specific operation is performed is displayed lighter than the previous image. Furthermore, it is also possible to store the range where the specific operation is performed a plurality of times, and display the image in an averaged range.

Example of Movement Instruction Control Performed by Processor81

Next, an example of the movement instruction control of the vehicle10performed by the processor81of the smartphone60will be described with reference toFIGS.6to14.

FIGS.6and7are flowcharts showing the exit instruction control of the vehicle10performed by the processor81of the smartphone60.FIGS.8to14are diagrams showing examples of the images displayed on the smartphone screen61of the smartphone60that performs the exit instruction control on the vehicle10.

The processing shown inFIG.6is started, for example, when the autonomous exit button (not shown) displayed on the smartphone screen61of the smartphone60is touched. The exit instruction control is executed in, for example, the situation as shown inFIG.4in which the user M attempts to cause the vehicle10to autonomously exit from the parking space P by using the smartphone60.

The processor81determines whether the autonomous exit instruction for starting the autonomous exit is received (step S11).

In step S11, when the autonomous exit instruction is not received (step S11: No), the processor81repeats the processing of step S11and waits until the autonomous exit instruction is received.

In step S11, when the autonomous exit instruction is received (step S11: Yes), the processor81displays on the smartphone screen61a first guidance image62that guides the autonomous exit instruction operation (step S12).

As shown inFIGS.8and9, the first guidance image62is displayed, for example, as a long horizontal bar extending in the left-right direction on the smartphone screen61. The up, down, left, and right directions of the smartphone screen61are directions based on, for example, orientations of characters displayed on the smartphone screen61. Below the first guidance image62on the smartphone screen61, a guide message63a(seeFIG.8) of operation guide which indicates that “please place finger on band (long horizontal bar)” and a guide message63b(seeFIG.9) of the operation guide which indicates that “vehicle moves when finger continuously moves upward and downward on screen” are alternately displayed. The first guidance image62is displayed substantially in a center of the smartphone screen61in the upper-lower direction. Further, above the first guidance image62on the smartphone screen61, an execution state image64that indicates an execution state of the exit instruction control for the vehicle10is displayed. An attention message65which indicates that “please directly check surroundings” for drawing attention of the user M who performs an instruction operation, and an interruption button66for interrupting the autonomous exit control are displayed on a right side of the execution state image64.

Next, the processor81determines whether the first guidance image62displayed in step S12is touched (step S13). Specifically, the processor81determines whether the finger is placed on the first guidance image62(long horizontal bar).

In step S13, if the first guidance image62is not touched (step S13: No), the processor81repeats the processing of step S13and waits until the first guidance image62is touched.

In step S13, if the first guidance image62is touched (step S13: Yes), the processor81displays a second guidance image67aor a second guidance image67bat the touched position (step S14).

As shown inFIGS.10and11, the second guidance images67aand67bare displayed, for example, as long horizontal bars shorter in the left-right direction than the first guidance image62at first positions62aand62btouched by the user M (placed with a finger) in the first guidance image62. The second guidance image67ashown inFIG.10is a guidance image that is displayed when the user M touches the first position62awhich is a portion on a right side of the first guidance image62. The second guidance image67bshown inFIG.11is a guidance image that is displayed when the user M touches the first position62bwhich is a portion on a left side of the first guidance image62.

Next, the processor81determines whether position information corresponding to the first positions62aand62btouched by the user M in the first guidance image62, that is, the positions where the second guidance images67aand67bare displayed is stored in the memory82(step S15).

In step S15, when the position information is not stored in the memory82(step S15: No), the processor81directly proceeds to processing of step S17.

In step S15, when the position information (step S15: Yes) is stored in the memory82, the processor81sets a reference position of the guidance image that prompts the specific operation based on the stored position information (step S16) and then directly proceeds to the processing of step S17.

Here, it is assumed that the autonomous exit of the vehicle10using the smartphone60is a first-time autonomous exit, and the position information related to the guidance image that prompts the specific operation is not stored in the memory82. Therefore, the processor81proceeds directly to the processing of step S17in step S15.

Next, the processor81displays third guidance images68aand68bdirected in first directions corresponding to the positions where the second guidance images67aand67bare displayed in step S14, that is, the positions touched by the user M (step S17).

The first direction corresponding to the position where the second guidance image67ais displayed is a direction corresponding to the first position62atouched by the user M at the right side in the first guidance image62, and for example, in the case shown inFIG.10, the first direction is a direction including a leftward direction symmetrical to the first position62aat the right side. Specifically, as in the example shown inFIG.10, the first direction is a direction toward the second position69aprovided in an area obliquely left below the first position62a. Similarly, the first direction corresponding to the position where the second guidance image67bis displayed is a direction corresponding to the first position62btouched by the user M at the left side in the first guidance image62, and for example, in the case shown inFIG.11, the first direction is a direction including a rightward direction symmetrical to the first position62bat the left side. Specifically, as in the example shown inFIG.11, the first direction is a direction toward the second position69bprovided in an area obliquely right below the first position62b.

As shown inFIG.10, the third guidance image68ais displayed, for example, as an arrow image pointing from the first position62aon the right side to the second position69a, which is an instruction position obliquely downward to the left. The third guidance image68ais displayed as an image that prompts the user M to perform the continuous position instruction operation in the first direction from the first position62ato the second position69a. The continuous position instruction operation is, for example, a swiping operation from the first position62ato the second position69a. Similarly, as shown inFIG.11, the third guidance image68bis displayed, for example, as an arrow image pointing from the first position62bon the left side to the second position69b, which is an instruction position obliquely downward to the right. The third guidance image68bis displayed as an image that prompts the user M to perform the continuous position instruction operation in the first direction from the first position62bto the second position69b. A guide message63cthat guides the continuous position instruction operation and indicates that “please move downward while touching screen” is displayed below each of the third guidance images68aand68b.

Accordingly, when the first position62aon the right side is touched, the second guidance image67ais displayed, and then the arrow image that guides the swiping operation to the lower left side is displayed. When the first position62bon the left side is touched, the second guidance image67bis displayed, and then the arrow image that guides the swiping operation to the lower right side is displayed. The second guidance images67aand67bdisplayed at the first positions62aand62bare guidance images that serve as starting points for the continuous position instruction operation (swiping operation).

Next, the processor81determines whether a first swiping operation is performed in the first directions from the positions where the second guidance images67aand67bare displayed, that is, in the directions indicated by the arrow images of the third guidance images68aand68b(step S18).

In step S18, when the first swiping operation is not performed (step S18: No), the processor81repeats the processing of step S18and waits until the first swiping operation is performed.

In step S18, when the first swiping operation is performed (step S18: Yes), the processor81displays a fourth guidance image70directed in second directions corresponding to the positions where the second guidance images67aand67bare displayed, that is, the positions touched by the user M (step S19).

For example, when the first swiping operation is performed in the first direction from the first position62awhere the second guidance image67ais displayed, as shown inFIGS.10and12, the second guidance image67adisplayed at the first position62ais moved to the second position69aobliquely downward to the left and displayed. Then, the fourth guidance image70is displayed in the second direction from the position of the second guidance image67amoved to the second position69aand displayed.

The second direction corresponding to the position where the second guidance image67ais displayed is a direction corresponding to the first position62atouched by the user M at the right side in the first guidance image62. The second direction is a direction different from the first direction from the first position62ato the second position69aobliquely downward to the left shown inFIG.10, that is, different from the direction indicated by the third guidance image68a.

For example, when the first swiping operation is performed in the first direction (direction toward second position69a) from the first position62awhere the second guidance image67ais displayed, as shown inFIG.12, the fourth guidance image70is displayed as an arrow image pointing, for example, to the right direction different from the first direction (direction indicated by third guidance image68a) from the position of the second guidance image67adisplayed at the second position69a. The fourth guidance image70is displayed as an image that prompts the user M to perform the continuous position instruction operation in the second direction different from the first direction. The continuous position instruction operation is, for example, a swiping operation from the second position69atoward the right direction. A guide message63dthat guides the continuous position instruction operation and indicates that “please move horizontally while touching screen” is displayed above the fourth guidance image70.

Although not shown, when the first swiping operation is performed in the first direction (direction toward second position69b) from the first position62bwhere the second guidance image67bis displayed, the second direction is a direction different from an obliquely downward right direction (first direction) indicated by the arrow image of the third guidance image68bshown inFIG.11. Then, the fourth guidance image70is displayed, for example, as an arrow image pointing to the left direction from the position of the second guidance image67bdisplayed at the second position69b.

In this way, when the first position62aon the right side is touched, the second guidance image67ais moved to the second position69aobliquely downward to the left and is displayed, and then the arrow image that guides the swiping operation to the right direction is displayed. When the first position62bon the left side is touched, the second guidance image67bis moved to the second position69bobliquely downward to the right and is displayed, and then the arrow image that guides the swiping operation to the left direction is displayed.

For example, the fourth guidance image70is displayed toward the second direction different from an instruction position of the swiping operation toward the first direction inFIG.10, that is, the first direction before a tip of the third guidance image68aindicated by the arrow image reaches an edge of the smartphone screen61(before protrudes from the smartphone screen61). The fourth guidance image70may be displayed toward the second direction different from the first direction when a distance of the instruction position of the swiping operation in the first direction, that is, a length of the third guidance image68aindicated by the arrow image exceeds a predetermined length.

Next, the processor81determines whether a second swiping operation is performed in the second directions from the second positions69aand69bwhere the second guidance images67aand67bare displayed, that is, in the direction indicated by the arrow image of the fourth guidance image70(step S20).

In step S20, when the second swiping operation is not performed (step S20: No), the processor81repeats the processing of step S20and waits until the second swiping operation is performed.

In step S20, when the second swiping operation is performed (step S20: Yes), the processor81displays a fifth guidance image71in rotation directions corresponding to the positions where the second guidance images67aand67bare first displayed, that is, the positions where the touch operation is started (step S21).

For example, the rotation direction corresponding to the position where the second guidance image67ais displayed is a rotation direction corresponding to the first position62atouched by the user M at the right side in the first guidance image62. When the first position62aon the right side is touched, the corresponding rotation direction becomes counterclockwise rotation. In this case, the smartphone screen61displays the counterclockwise rotation fifth guidance image71. Specifically, as shown inFIG.13, when the first position62aon the right side is touched, the fifth guidance image71in the counterclockwise rotation state indicated by arrows72is displayed. For example, the fifth guidance image71is displayed as if six spheres are rotating counterclockwise. The fifth guidance image71is displayed in a manner of rotating from the position of the second guidance image67adisplayed at the second position69a. The fifth guidance image71is displayed as a rotation guidance image that prompts the user M to perform a rotation instruction operation that causes the counterclockwise rotation. The rotation instruction operation that causes the counterclockwise rotation is, for example, a rotation swiping operation in which one or more fingers slide counterclockwise. A guide message63ethat guides the rotation instruction operation and indicates that “please turn while touching screen” is displayed above the fifth guidance image71.

Although not shown, when the first position62bon the left side is touched, the corresponding clockwise rotation fifth guidance image71is displayed. Specifically, the fifth guidance image71is displayed in a manner that the six spheres begin to rotate clockwise from the position of the second guidance image67bdisplayed at the second position69binFIG.11.

Accordingly, when the first position62aon the right side is touched, a plurality of counterclockwise rotation spherical images that guide a counterclockwise rotation swiping operation from the position of the second guidance image67amoved to the second position69aare displayed. When the first position62bon the left side is touched, a plurality of clockwise rotation spherical images that guide a clockwise rotation swiping operation from the position of the second guidance image67bmoved to the second position69bare displayed. The second guidance images67aand67bdisplayed at the first positions62aand62bare rotation guidance images that serve as starting points for the rotation instruction operation (rotation swiping operation).

Next, the processor81proceeds to processing shown inFIG.7and determines whether the rotation swiping operation is started (step S22).

In step S22, when the rotation swiping operation is not started (step S22. No), the processor81repeats the processing of step S22and waits until the rotation swiping operation is started.

In step S22, when the rotation swiping operation is started (step S22: Yes), the processor81hides the fifth guidance image71displayed on the smartphone screen61, and displays an instruction icon73that moves to follow the instruction position of the rotation swiping operation of the user M (step S23).

For example, as shown inFIG.14, the fifth guidance image71(seeFIG.13) that is displayed using six rotation spheres is hidden, and the instruction icon73representing the autonomous exit instruction of the vehicle10is displayed. The instruction icon73moves to follow the touch position of the user M on the smartphone screen61. The instruction icon73is an instruction icon in which the second guidance image67adisplayed inFIG.13moves to follow the finger movement of the rotation swiping operation of the user M. In addition, a guide message63ffor stopping the autonomous exit of the vehicle10is displayed, which indicates that “release the finger to stop”. Accordingly, when the rotation swiping operation is performed on the smartphone screen61, the instruction icon73rotationally moves to follow the touch position, and the vehicle10starts to move according to the rotational movement. In addition, when the finger that touches the smartphone screen61is released from the smartphone screen61, the movement of the vehicle10stops.

Next, the processor81transmits an exit starting signal to the vehicle10to start the autonomous exit of the vehicle10(step S24). Accordingly, the autonomous parking control unit55of the vehicle10starts the autonomous exit of the vehicle10according to the exit starting signal transmitted from the smartphone60. When the autonomous exit of the vehicle10is started, as shown inFIG.14, an image corresponding to the movement of the vehicle10, that is, the execution state image64of the vehicle10exiting from the parking space P is displayed.

Next, the processor81determines whether the rotation swiping operation on the smartphone screen61is stopped (step S25).

In step S25, when the rotation swiping operation is stopped (step S25: Yes), the processor81transmits an exit stop signal to the vehicle10to stop the autonomous exit of the vehicle10(step S26).

Next, the processor81determines whether the rotation swiping operation on the smartphone screen61is restarted (step S27).

In step S27, when the rotation swiping operation is not restarted (step S27: No), the processor81waits until the rotation swiping operation is restarted.

In step S27, when the rotation swiping operation is restarted (step S27: Yes), the processor81restarts the instruction control of the autonomous exit of the vehicle10(step S28), returns to step S23, and repeats the processing.

Meanwhile, in step S25, when the rotation swiping operation is not stopped (step S25: No), the processor81determines whether the autonomous exit of the vehicle10is completed (step S29). Completion of the autonomous exit of the vehicle10can be recognized based on, for example, an exit completion signal transmitted from the autonomous parking control unit55of the vehicle10. The autonomous exit is completed when the vehicle10moves out of the parking space P, for example.

In step S29, when the autonomous exit of the vehicle10is not completed (step S29: No), the processor81returns to step S25and repeats the processing.

In step S29, when the autonomous exit of the vehicle10is completed (step S29: Yes), the processor81stores the position information of the swiping operation performed by the user M in the memory82(step S30), and ends the processing of the exit instruction control. The position information of the swiping operation performed by the user M includes information of the touch position of the first guidance image62touched by the user M in step S13, the position information of the first swiping operation performed by the user M in step S18, the position information of the second swiping operation performed by the user M in step S20, and the position information of the rotation swiping operation performed by the user M in step S22. The processor81stores in the memory82the position information indicating where on the smartphone screen61the user M performs the swiping operation. In the middle of the series of processing described above, for example, when the user M touches the interruption button66during the series of processing described above, the processor81ends the processing of the exit instruction control.

<Second-Time and Subsequent-Time Movement Instruction Control>

Next, the exit instruction control for second-time and subsequent-time autonomous exit using the smartphone60will be described with reference toFIGS.15to19and flowcharts ofFIGS.6and7.FIGS.15to19are diagrams showing examples of images displayed on the smartphone screen61of the smartphone60, and correspond toFIGS.10to14in the above-described first-time exit instruction control.

In cases of second-time and subsequent-time exit instruction control, the processing from step S11to step S16inFIG.6is executed in the same manner as the processing from step S11to step S16in the above-described first-time exit instruction control. In the following description, it is assumed that the exit instruction control is the second-time and subsequent-time exit instruction control.

For example, it is assumed that the first positions62aand62bof the first guidance image62touched by the user M during the second-time autonomous exit are the same side positions as the first positions62aand62bof the first guidance image62touched by the user M in the first-time autonomous exit. Specifically, it is assumed that the user M touches the first position62aon the right side of the first guidance image62both during the first-time autonomous exit and during the second-time autonomous exit, or that the user M touches the first position62bon the left side of the first guidance image62both during the first-time autonomous exit and during the second-time autonomous exit.

Since the position information of the first-time swiping operation performed by the user M is stored in the memory82(namely, the position information is in the memory82), the processor81proceeds to step S16in branch processing of step S15, and sets the reference position of the guidance image that prompts the specific operation based on the stored position information of the first-time swiping operation. The reference position of the guidance image is a position that serves as a display reference for the guidance images (third guidance images68aand68b, fourth guidance image70, and fifth guidance image71) displayed on the smartphone screen61. The reference position of the guidance image includes, for example, positions corresponding to the second position69ashown inFIGS.10,12, and13and the second position69bshown inFIG.11in the first-time exit instruction control.

For example, when the position of the first swiping operation performed by the user M in step S18in the first-time exit instruction control is a position at the right side of the third guidance image68a(seeFIG.10) displayed on the smartphone screen61, the reference position of the guidance image set in step S16in the second-time exit instruction control, that is, a second position169a, is set to a position at the right side of the second position69ashown inFIG.10. Similarly, when the position of the first swiping operation performed by the user M in step S18in the first-time exit instruction control is a position at the left side of the third guidance image68b(seeFIG.11) displayed on the smartphone screen61, the reference position of the guidance image set in step S16in the second-time exit instruction control, that is, a second position169b, is set to a position at the right side of the second position69bshown inFIG.11.

Next, the processor81displays third guidance images168aand168bdirected in the first directions corresponding to the positions where the second guidance images67aand67bare displayed in step S14, that is, the positions touched by the user M, and directed to the second position169aand169bon the right side and the left side set in step S16(step S17).

The third guidance image168awhich is directed to the second position169aon the right side in the first direction corresponding to the position where the second guidance image67ais displayed is, for example, the third guidance image168ashown inFIG.15. As shown inFIG.15, the third guidance image168ais displayed toward the second position169adisplayed at the position on the right side of the position of the second position69ashown inFIG.10. The third guidance image168bwhich is directed to the second position169bon the left side in the first direction corresponding to the position where the second guidance image67bis displayed is, for example, the third guidance image168bshown inFIG.16. As shown inFIG.16, the third guidance image168bis displayed toward the second position169bdisplayed at a position on the left side of the position of the second position69bshown inFIG.11.

Next, the processor81determines whether the first swiping operation in the first directions indicated by the third guidance images168aand168bis performed (step S18), when the first swiping operation is performed (S18: Yes), the processor81displays a fourth guidance image170directed in the second direction, starting from a position of a second guidance image167adisplayed after being moved to the second position169aor169bon the right side or the left side (step S19).

Specifically, when the first swiping operation is performed in the first direction (direction toward second position169a) from the first position62awhere the second guidance image67ais displayed, as shown inFIG.17, the fourth guidance image170is displayed as an arrow image pointing, for example, to the right direction different from the first direction (direction indicated by third guidance image168a), starting from the position of the second guidance image167adisplayed at the second position169aon the right side. The fourth guidance image170is displayed on the right side of the fourth guidance image70shown inFIG.12.

Although not shown, when the first swiping operation is performed in the first direction (direction toward second position169b) from the first position62bwhere the second guidance image67bis displayed, the fourth guidance image170is displayed as an arrow image pointing, for example, to the left direction, starting from the position of the second guidance image displayed at the second position169bon the left side.

In this case, for example, when the position of the second swiping operation performed by the user M in step S20in the first-time exit instruction control is a position at a right side or a left side of the position of the fourth guidance image70displayed inFIG.12, the position indicated by the fourth guidance image170inFIG.17displayed in step S19may be displayed to be the right side or the left side of the position indicated by the fourth guidance image70shown inFIG.12.

Next, the processor81determines whether the second swiping operation in the second direction indicated by the fourth guidance image170is performed (step S20). When the second swiping operation is performed (S20: Yes), the processor81displays a fifth guidance image171in the rotation direction corresponding to the position where the touch operation is started, starting from the position of the second guidance image167adisplayed at the second position169aor169bon the right side or the left side (step S21).

Specifically, when the second swiping operation is performed in the second direction from the right second position169awhere the second guidance image167ais displayed, as shown inFIG.18, the fifth guidance image171is displayed as a guidance image that guides a counterclockwise rotation swiping operation as indicated by arrows172starting from the position of the second guidance image167adisplayed at the second position169aon the right side. The fifth guidance image171is displayed on a right side of the fifth guidance image71shown inFIG.13.

Although not shown, when the second swiping operation is performed in the second direction from the second position169bon the left side where the second guidance image is displayed, the fifth guidance image171is displayed as a clockwise rotation guidance image starting from the position of the second guidance image displayed at the second position169bon the left side.

In this case, for example, when the position of the rotation swiping operation performed by the user M in step S22in the first-time exit instruction control is a position on the right side or the left side of the position of the rotation swiping operation displayed inFIG.13, the rotation position of the fifth guidance image171ofFIG.18displayed in step S21may be displayed to be the right side or the left side of the rotation position of the fifth guidance image71shown inFIG.13.

Next, the processor81proceeds to the processing shown inFIG.7, determines whether the rotation swiping operation is started (step S22), and when the rotation swiping operation is started (step S22: Yes), the processor81hides the fifth guidance image171, and then displays an instruction icon173that moves to follow the instruction position of the rotation swiping operation of the user M, starting from the position of the second guidance image167adisplayed at the second positions169aor169bon the right side or the left side (step S23).

Specifically, as shown inFIG.19, a swiping frame174for performing the rotation swiping operation is displayed starting from the position of the second guidance image167adisplayed at the second position169aon the right side, and the instruction icon173is displayed within the swiping frame174. The swiping frame174and the instruction icon173are displayed on a right side of the positions of the swiping frame74and the instruction icon73shown inFIG.14.

Next, the processor81proceeds to the processing of step S22and causes the vehicle10to start the autonomous exit. The processing from step S22to step S29is executed in the same manner as the processing from step S22to step S29in the above-described first-time exit instruction control. In step S30, the processor81stores in the memory82the position information of the swiping operation performed by the user M in the second-time exit instruction control, and ends the processing of the exit instruction control.

In the above-described example of the exit instruction control, the second-time exit instruction control has been described, but the same control is performed in third-time and subsequent-time exit instruction control. In the second-time exit instruction control, the guidance image that prompts the specific operation is displayed based on the position information of the swiping operation stored in the memory82in the first-time exit instruction control, but in a case of the third-time and subsequent-time exit instruction control, the present invention is not limited thereto. For example, in the third-time exit instruction control, the guidance image may be displayed based on the previous position information, or the guidance image may be displayed based on average position information in the position information for the past several times.

As described above, the processor81of the smartphone60causes the smartphone screen61to display a guidance image that prompts a specific operation (swiping operation, rotation swiping operation) when waiting for reception of the specific operation, and stores position information of the specific operation performed by the user M based on the guidance image. When the position information is stored at the time of waiting for reception of the specific operation, the guidance image that prompts the specific operation is displayed based on the position information on the smartphone screen61. Accordingly, the guidance image can be displayed based on the position of the operation accepted as the specific operation in the swiping operation performed by the user M in the past. Therefore, the user M can be guided to an operation with less burden, and usability can be improved.

In a case where the position information is stored (namely, if the position information is in the memory82) when waiting for reception of the specific operation, the processor81displays the guidance image that prompts the specific operation and instructs the position based on the position information on the smartphone screen61. Accordingly, the specific operation can be performed according to the guidance image that instructs the operation position, and thus, the user M can be guided to an operation with less burden, and the usability is improved.

The processor81of the smartphone60stores the position information for each of the rotation instruction operation in the clockwise direction and the rotation instruction operation in the counterclockwise direction. Accordingly, even when the user M performs the rotation instruction operation in either the clockwise direction or the clockwise direction (right hand or left hand), the rotation guidance image based on the stored position information of the rotation instruction operation can be displayed on the smartphone screen61. Therefore, the user M can be guided to the swiping operation with less burden.

<Modification of Movement Instruction Control>

Next, modifications of the movement instruction control of the vehicle10performed by the processor81of the smartphone60will be described with reference toFIGS.20to23.

<First Modification>

FIG.20is a diagram showing the third guidance image displayed on the smartphone screen61of the smartphone60in a first modification of the movement instruction control. The image inFIG.20corresponds to the images inFIGS.10and15in the above embodiment. As shown inFIG.20, when the first position62aon the right side of the first guidance image62is touched by the user M, the processor81displays the third guidance image68adirected from the second guidance image67adisplayed at the first position62ato the second position69a, and superimposes on the guidance image and displays an operation range image181indicating a range where the user M performs the specific operation. The specific operation includes the swiping operation and the rotation swiping operation as described above.

In the example shown in the drawing, the operation range image181is displayed as an image with a shaded area surrounding the second guidance image67aand the third guidance image68a. The operation range image181is an image showing the range where the specific operation is performed during the previous autonomous exit. Therefore, for example, when the user M performs the specific operation in the range on the right side of the position of the guidance image displayed on the smartphone screen61during the previous autonomous exit, the operation range image181is displayed on the right side position of the smartphone screen61. In this regard, when the user M performs the specific operation in the range on the left side of the position of the guidance image displayed on the smartphone screen61during the previous autonomous exit, the operation range image181is displayed on the left side position of the smartphone screen61. The example shown in the drawing shows a state where the operation range image181is displayed at the right side position.

Accordingly, according to the processor81of the first modification, when the position information is stored at the time of waiting for reception of the specific operation, the smartphone screen61is caused to display the guidance image that prompts the specific operation and on which an image that indicates the range where the specific operation is performed is superimposed. Accordingly, the specific operation can be performed according to the guidance image that instructs the operation range, and thus, the user M can be guided to an operation with less burden, and the usability is improved.

<Second Modification>

FIG.21is a diagram showing the third guidance image68aand an operation range image182displayed on the smartphone screen61of the smartphone60in a second modification of the movement instruction control. As shown inFIG.21, the processor81displays the operation range image182superimposed on the third guidance image68aby changing a display state according to the number of times of receptions of the specific operation. Specifically, the user M performs the specific operation in the range on the right side of the position of the guidance image displayed on the smartphone screen61, and the processor81lightly displays the operation range image182when the number of times of the specific operations on the right side exceeds a constant value. The display of the operation range image182may be hidden when the number of times of receptions exceeds a predetermined number. Accordingly, the operation range image can be prevented from being conspicuously displayed even though the operation range image that indicates the range where the user performs the specific operation in the past is unnecessary.

<Third Modification>

FIG.22is a diagram showing the third guidance image68aand operation range images183and184displayed on the smartphone screen61of the smartphone60in a third modification of the movement instruction control. As shown inFIG.22, the processor81changes a display mode of the operation range images183and184superimposed on the third guidance image68aand displayed according to the number of times of reception of the specific operation (before and after reception). Specifically, the processor81displays the range where the specific operation is performed during the previous-time autonomous exit with the dark operation range image183, displays, with the light operation range image184, the range where the specific operation is performed during the second previous-time autonomous exit, and hides the operation range image of the range where the specific operation is performed during third-time or more previous-time autonomous exit. Accordingly, a tendency of the specific operation of the user M can be reflected and the operation range image can be displayed, and thus, the usability is improved. The display mode may be displayed according to, for example, a level of a display transparency of the image, in addition to the dark and light display.

<Fourth Modification>

FIG.23is a diagram showing the third guidance image168aand the operation range image181displayed on the smartphone screen61of the smartphone60in a fourth modification of the movement instruction control. The image inFIG.23is an image obtained by combining the third guidance image168ainFIG.15in the above embodiment and the operation range image181inFIG.20in the first modification. As shown inFIG.23, the processor81displays the third guidance image168adirected toward the second position169aon the right side, and superimposes and displays the operation range image181on the right side which indicates the range where the user M performs the specific operation previous time. Accordingly, by displaying the combination of the guidance image and the operation range image, the user M can be guided to an operation with less burden, and the usability is improved.

Although the cases where the operation range image is superimposed on the third guidance image has been described in the first to fourth modifications, the present invention is not limited thereto. For example, the operation range image may be superimposed on the fourth guidance image inFIG.12, the fifth guidance image inFIG.13, and the instruction icon inFIG.14.

In the movement instruction control performed by the above processor81, the autonomous exit of the vehicle10has been described, but the movement instruction control can also be applied to the autonomous parking of the vehicle10, for example.FIG.24is a diagram showing a situation where the user M of the vehicle10issues a parking instruction to autonomously park the vehicle10in the parking space P from the outside of the vehicle10by using the smartphone60carried by the user M.

FIG.25shows an example of an image including the first guidance image displayed on the smartphone screen61when the processing of the exit instruction control shown inFIGS.6and7is applied to the parking instruction control, and an instruction to start the autonomous parking is received in steps S11and S12. As shown inFIG.25, the smartphone screen61displays the first guidance image62touch-operated by the user M, and also displays the vehicle10stopped in front of the parking space P and waiting for the autonomous parking control.

Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and modifications, improvements, and the like can be made as appropriate.

For example, in the above-described embodiment, an example in which the moving object is a vehicle (four-wheeled automobile) has been described, but the present invention is not limited thereto. For example, the moving object may be a two-wheeled vehicle or a Segway. Further, the concept of the present invention can be applied not only to a vehicle but also to a robot, a ship, an aircraft, or the like that is provided with a drive source and is movable according to power of the drive source.

The control method described in the above embodiment can be implemented by executing a control program prepared in advance on a computer. The control program is recorded in a computer-readable storage medium and is executed by being read from the storage medium. In addition, the control program may be provided in a form of being stored in a non-transitory storage medium such as a flash memory, or may be provided via a network such as the Internet. The computer that executes the control program may be provided in a control device, may be provided in an electronic device such as a smartphone, a tablet terminal, or a personal computer capable of communicating with the control device, or may be provided in a server device capable of communicating with the control device and the electronic device.

In addition, at least the following matters are described in the present specification. Although the corresponding components or the like in the above embodiment are illustrated in parentheses, the present invention is not limited thereto.(1) A portable information terminal (smartphone60) for a user (user M) of a moving object (vehicle10), the information terminal including:a display (smartphone screen61) configured to perform a position instruction operation on a display image; anda controller (processor81) configured to perform movement control of the moving object based on a specific operation of the user on the display, in whichthe controller is configured to:cause the display to display a guidance image that prompts the specific operation when waiting for receiving the specific operation,store position information of the specific operation performed by the user based on the guidance image, anddisplay the guidance image that prompts the specific operation based on the position information on the display in a case of storing the position information when waiting for receiving the specific operation.

According to (1), the guidance image is displayed based on the position of the operation performed by the user in the past and accepted as the specific operation, so that an operation with less burden for the user can be guided, and usability is improved.(2) The information terminal according to (1), in whichif the controller stores the position information when waiting for receiving the specific operation, the controller displays the guidance image that prompts the specific operation and indicates a position based on the position information on the display.

According to (2), the guidance image that indicates the position where the specific operation is performed is displayed, so that an operation with less burden for the user can be guided, and the usability is improved.(3) The information terminal according to (1) or (2), in whichthe position information indicates a range where the user performs the specific operation on the display, andif the controller stores the position information w % ben waiting for receiving the specific operation, the controller causes the display to display the guidance image that prompts the specific operation and on which an image that indicates the range is superimposed.

According to (3), the guidance image that indicates the range where the specific operation is performed is displayed, so that an operation with less burden for the user can be guided, and the usability is improved.(4) The information terminal according to (3), in whichif the controller stores the position information when waiting for receiving the specific operation, the controller changes the image that indicates the range based on a history of receiving the specific operation from the user.

According to (4), a guidance image that is highly effective for the user can be displayed.(5) The information terminal according to any one of (1) to (4), in whichthe specific operation includes a rotation instruction operation,the guidance image includes a rotation guidance image that prompts the rotation instruction operation, andthe controller stores the position information for each of the rotation instruction operation in a clockwise direction and the rotation instruction operation in a counterclockwise direction.

According to (5), even when the user M performs the rotation instruction operation in either the clockwise direction or the counterclockwise direction (right hand or left hand), the rotation guidance image based on the stored position information of the rotation instruction operation can be displayed, and an operation with less burden for the user can be guided, and thus usability is improved.(6) A control method performed by a controller of a portable information terminal for a user of a moving object, the information terminal including: a display configured to perform a position instruction operation on a display image; and the controller configured to perform movement control of the moving object based on a specific operation of the user on the display, in whichthe control method includes:cause the display to display a guidance image that prompts the specific operation when waiting for receiving the specific operation,store position information of the specific operation performed by the user based on the guidance image, anddisplay the guidance image that prompts the specific operation based on the position information on the display in a case of storing the position information when waiting for receiving the specific operation.

According to (6), the guidance image is displayed based on the position of the operation performed by the user in the past and accepted as the specific operation, so that an operation with less burden for the user can be guided, and the usability is improved.(7) A non-transitory computer-readable recording medium storing a control program for causing a controller of a portable information terminal for a user of a moving object to execute processing, the information terminal including: a display configured to perform a position instruction operation on a display image; and the controller configured to perform movement control of the moving object based on a specific operation of the user on the display, in which the processing includes:causing the display to display a guidance image that prompts the specific operation when waiting for receiving the specific operation,storing position information of the specific operation performed by the user based on the guidance image, anddisplaying the guidance image that prompts the specific operation based on the position information on the display in a case of storing the position information when waiting for receiving the specific operation.

According to (7), the guidance image is displayed based on the position of the operation performed by the user in the past and accepted as the specific operation, so that an operation with less burden for the user can be guided and usability is improved.