Source: https://patents.justia.com/patent/20100277438
Timestamp: 2019-09-17 12:25:05
Document Index: 562314155

Matched Legal Cases: ['art 12', 'art 12', 'art 12', 'art 12', 'art 12', 'art 12', 'art 12', 'art 12', 'art 12', 'art 12', 'art 12', 'art 12']

US Patent Application for Operation apparatus for in-vehicle electronic device and method for controlling the same Patent Application (Application #20100277438 issued November 4, 2010) - Justia Patents Search
Justia Patents Including Optical DetectionUS Patent Application for Operation apparatus for in-vehicle electronic device and method for controlling the same Patent Application (Application #20100277438)
Operation apparatus for in-vehicle electronic device and method for controlling the same
Apr 29, 2010 - DENSO CORPORATION
As follows, an embodiment will be described with reference to drawings. FIG. 1 is a perspective view showing one example of an operation apparatus for an in-vehicle electronic device applied with an image display device. The operation apparatus 1 is located in an interior of a vehicle such as an automobile. The operation apparatus 1 includes a monitor device (display device) 15 located in a center portion of an instrument panel and an operation unit (input unit, touch panel, touch input device) 12 located in a center console C such that an occupant can manipulate the operation unit 12 from either a driver seat 2D or a passenger seat 2P. The purpose of the operation unit 12 is not limited. For example, an occupant may use the operation unit 12 for manipulating a car navigation device and/or a car audio device while looking at a screen of the monitor device 15 provided in the center console. The operation unit 12 is mounted so that an input surface of the operation unit 12 is directed upward. The input surface of the operation unit 12 may include, for example, a generally-known touch panel 12a having a resistive film type structure, a surface acoustic wave type structure, or a capacitance type structure. The touch panel 12a is configured as a transparent instruction panel having a base material of a transparent resin sheet or a glass sheet. The touch panel 12a has an upper surface for receiving a touch operation of a fingertip of an operator. An input coordinate system is set on the sheet surface correspondingly to the screen area of the monitor device 15.
FIG. 2 is a sectional view showing an internal structure of the input unit 12. The touch panel 12a is fitted in an upper surface of a case 122e so that the input surface 102b is located at the top side. A hand imaging camera (imaging device) 12b, an illumination light source 12c, and an imaging optical system are accommodated in a case 12d and configured as an image obtaining unit. The illumination light source 12c is, for example, a light emitting diode configured as a monochromatic light source. The illumination light source 12c has a convex-shaped mold to enhance a directivity of luminescence toward an upper side of the illumination light source 12c. The illumination light source 12c includes multiple light source elements to surround a lower surface of the touch panel 12a. Each element of the illumination light source 12c has a molded tip end, which causes high luminescence, inclined and directed toward the lower surface of the touch panel 12a. The illumination light source 12c is mounted so that a palm surface of a hand H of an operator on the input surface 102b causes a primary image reflected light RB1 downward through the touch panel 12a. The imaging optical system has a first reflection part 12p and a second reflection part 12r. The first reflection part 12p is configured as a prism plate 12p located directly under the touch panel 12a and opposed to the touch panel 12a. The prism plate 12p is a transparent plate having a surface defining minute triangular-column-shaped prisms arranged in parallel with each other. The first reflection part 12p reflects the primary image reflected light RB1 of the operator's hand H obliquely upward, thereby to cause a secondary image reflected light RB2 outside of an opposed space 12f between the prism plate 12p and the touch panel 12a. The second reflection part 12r is a flat reflector 12r located on the lateral side of the opposed space 12f. The second reflection part 12r is located outside of the opposed space 12f. The second reflection part 12r reflects the secondary image reflected light RB2 toward the lateral side, thereby to cause a tertiary image reflected light RB3 on the hand imaging camera 12b. The hand imaging camera 12b is located on the opposite side of the second reflection part 12r through the opposed space 12f. The hand imaging camera 12b is provided in a position corresponding to a focus position of the tertiary image reflected light RB3 for obtaining an image of the hand H including an operator's finger.
As shown in the enlarged view in FIG. 2, the prism plate 12p includes minute rib-shaped prisms arranged along a mirror reference surface MBP. The minute prisms of the prism plate 12p are arranged close to each other and in parallel with each other. Each of the minute prisms has a reflective surface inclined at a constant angle relative to the mirror reference surface MBP. The prism plate 12p is configured to reflect light, which comes in its normal-line direction, obliquely toward the lateral side even when the prism plate 12p is not inclined. Therefore, the first reflection part 12p for causing lateral reflection can be located at the lower side of the touch panel 12a to be opposed to the touch panel 12a and in parallel with the touch panel 12a. Thus, the size of the first reflection part 12p in the vertical direction size of the opposed space 12f can be significantly reduced.
In addition, the opposed space 12f is interposed, between the second reflection part 12r and the hand imaging camera 12b, which are opposed to each other. Thereby, the tertiary image reflected light RB, which directly enters into the hand imaging camera 12b can be directed so as to pass through the opposed space 12f. In the present structure, the second reflection part 12r and the hand imaging camera 12b can be located close to the lateral sides of the touch panel 12a. In addition, the incidence path of the image reflected light, which comes from the hand H to the hand imaging camera 12b, is folded into three within the opposed space 12f. Therefore, the entire optical imaging system can be significantly downsized, and the thickness of the case 12d can be also reduced. In particular, the input unit 12 can be significantly thin-shaped and reduced in the size by reducing the size of the input surface 102b of the touch panel 12a in the vertical and horizontal directions. Thus, the input unit 12 can be mounted to a vehicle, in which the width of the center console section C (FIG. 1) is relatively small, or a vehicle in which only a limited mounting space can be secured on the front side of a shift lever.
The input surface 102b of the touch panel 12 corresponds to an imaging area 102b of the hand imaging camera 12b. The size of the input surface 102b in the vertical direction (Y-direction) is set to, for example, 60 to 90 mm (75 mm as one example), such that a part of a fingertip of a middle finger in the longitudinal direction is within the input surface 102b when a hand of an average adult person is assumed. In the present structure, only a portion of a finger on the tip side from the base of the finger is indicated in the screen area of the monitor device 15. Therefore, a palm portion other than the finger does not exert influence to the indication of the screen area. Thus, a below-mentioned indication operation using an instruction object image can significantly simplified. Furthermore, the size of the input surface 102b in the horizontal direction (X-direction) is, for example, 110 to 130 mm (120 mm as one example). When the hand is put above the input surface 102b, and when fingers of the hand are widely unbent, the index finger, the middle finger, the third finger, and the little finger of the hand are in the imaging area, and the thumb of the hand is outside of the imaging area. When fingers are suitably put close to each other, all the fingers may be put within the imaging area.
FIG. 3 is a block diagram showing an electronic connection among components of the operation apparatus 1. An operation ECU 10 functions as a main controller of the operation apparatus 1. The operation ECU 10 is a computer device mainly configured of a CPU 101. Specifically, the operation ECU 10 has a structure in which the CPU 101, a RAM 102, a ROM 103, a graphics controller 110, a video interface unit 112, a touch-panel interface unit 114, a general-purpose input and output (I/O) unit 104, and a serial communication interface unit 116 are interconnected with each other via an internal bus 105. The graphics controller 110 is connected with a display video RAM 111 and the monitor device 15. The video interface unit 112 is connected with an imaging video RAM and the hand imaging camera 12b. The touch panel 12a is connected with the touch-panel interface unit 114. The general-purpose input and output unit 104 is connected with the illumination light source 12c via a driver (driver circuit) 115. The serial communication interface unit 116 is connected with an in-vehicle serial communication bus 30 such as a CAN communication bus. The in-vehicle serial communication bus 30 is configured to intercommunicate with another ECU, such as a navigation ECU 200 for controlling a navigation device and a body ECU 300, connected via network communications.
The video interface unit 112 regularly receives an analog or digital image signal obtained by the hand imaging camera 12b. The image signal is stored as image frame data in an imaging video RAM 113 at a predetermined time interval. A memory content of the imaging video RAM is arbitrary updated when receiving new image frame data.
The touch-panel interface unit 114 includes a drive circuit suitable for the type of the touch panel 12a. In addition, the touch-panel interface unit 114 detects an inputted position (inputted touch point) specified by a touch operation to the input surface 102b based on a signal inputted from the touch panel 12a. The touch-panel interface unit 114 outputs a detection result of the inputted position as inputted position coordinate information.
A two-dimensional coordinates correspondence is determined among the imaging area of the hand imaging camera 12b, the input surface of the touch panel 12, and the screen area of the monitor device 15. More specifically, a two-dimensional coordinates correspondence is determined among an image obtained by the hand imaging camera 12b, the input surface of the touch panel 12, and the input screen image frame data and the instruction object image frame data, which determine a content indicated on the monitor device 15. The ROM 103 stores the following software to be executed by the CPU 101.
Touch-Panel Control Software 103a:
The touch-panel control software 103a obtains the coordinates of the inputted touch point specified by a touch operation via the touch-panel interface unit 114. The touch-panel control software 103a further obtains reference information for determining an operated content and an inputted content. The reference information is sent from the navigation ECU 200 together with the input screen image frame data. The reference information includes, for example, region specification information on a soft button and an operated command content to be outputted when the soft button is touch-operated. The touch-panel control software 103a determines the present operated content and the present inputted content based on the obtained touch point coordinates and the obtained reference information. Thus, the touch-panel control software 103a outputs an execution instruction corresponding to the content to the navigation ECU 200.
Display Control Software 103b:
The display control software 103b outputs an import instruction to cause the graphics controller 110 to import the input screen image frame data. The display control software 103b further transfers instruction object image frame data, which is produced by a below-mentioned method, to the graphics controller 110.
Fingertip Point Computation Software 103c:
The fingertip point computation software is executed in a work area of a fingertip point processing memory 1102a of the RAM 1102. The fingertip point computation software functions as a tip end extracting unit, an image tip end position specifying unit, and a fingertip determination unit. The fingertip point computation software performs a computation to binarize a photographic image of an operator's hand obtained by the hand imaging camera 12b and performs a computation to specify a fingertip position in the actually obtained finger image of the hand. More specifically, the fingertip point computation software performs a computation to specify a predetermined representative point of a tip end region ta as an imaged fingertip point tp in an insertion direction of the binarized photographic image. In this case, the fingertip point computation software specifies a geometric centroid position as the predetermined representative point. The display control software 103b further determines whether the imaged fingertip point tp really specifies the fingertip point tp based on at least the size of the tip end region ta and the area of the tip end region ta. An output device of the video interface unit may include a binarizing circuit for binarizing pixels of an image, and the photographic image of the hand may be binarized by the binarizing circuit beforehand.
Instruction Object Image Composition Software 103d:
The instruction object image composition software 103d is executed in a work area of an instruction object image composition memory 1102b of the RAM 1102. The instruction object image composition software 103d functions as an instruction object image indication unit. Specifically, as shown in FIG. 9, the instruction object image composition software 103d may compose the instruction object image by superposing a hand image SF′ on the instruction object image and may indicate the superimposed image on the screen of the monitor device 15. Alternatively, as shown in FIG. 10, the instruction object image composition software 103d may compose the instruction object image by generating an instruction object image F1 of a finger, which has the width smaller than the width of the finger of the hand image, based on instruction object image data 103e generated separately from the photographic image. Thus, the instruction object image composition software 103d may superimpose the generated instruction object image F1 on the instruction object image and may indicate the superimposed image on the screen of the monitor device 15. In this case, the instruction object image composition software 103d performs a processing to put the instruction object image F1 on the instruction object image frame so that the fingertip position of the instruction object image F1 coincides with the imaged fingertip point.
User Interface Engine 103e:
The user interface engine 103e determines a prescribed interface specification, which is a relationship between an instruction input state, which is specified by at least one of the inputted touch point and the imaged fingertip point on the operation panel, and an operation instruction output content outputted to an in-vehicle electronic device such as a car navigation system and a car audio system. In addition, the user interface engine 103e outputs operation instruction information according to the interface specification and the instruction input state on the operation panel. The user interface engine 103e includes an interface specification modifying unit. The interface specification modifying unit has a function to arbitrarily modify the interface specification according to at least a traveling state of a vehicle (in particular, a vehicle speed), the number of an imaged fingertip point, determination result whether multiple touch operations on the touch input device 12 are simultaneously detected when an imaged fingertip point is detected, and the like.
As follows, a method for specifying the imaged fingertip point of the hand by using the operation apparatus 1 will be described with reference to a flow chart. The fingertip point computation software 103c performs a main operation of the present method. It is assumed that the screen (region) of the monitor device 15 indicates a keyboard input screen of FIG. 10 excluding the hand image SF, in response to a command input such as a touch input operation on another screen. It is noted that another input screen such as a map screen may be indicated in this case.
FIG. 5 shows a fingertip point specifying operation repeatedly executed at a constant cycle. In the above-described state, as shown in FIG. 2, when the hand H is moved close to the input surface 102b of the touch panel 12a, the hand imaging camera 12b obtains an image of the hand according to the reflected light of the hand caused by the illumination light emitted from the illumination light source 12c. At S101 of FIG. 5, the photographic image (hand image) of the hand H is obtained. A pixel corresponding to the hand image appears lighter than a pixel corresponding to the background region due to a reflected light. The luminosity of a pixel of the hand image and the luminosity of a pixel of the background region are compared with a suitable threshold and binarized. Thus, as shown in the image A of FIG. 4, image separation can be carried out to generate an image object region (black region in the image A) showing a high-intensity pixel value of 1 and a background region (white region in the image A) showing a low-intensity pixel value of 0. At S102, the presently binarized image data is stored as first image data A.
At S103, an area rate σ of the image object region in the first image data A is calculated. When an image object does not exist in the imaging area of the hand imaging camera 12b, the area rate σ of the image object region becomes less than or equal to a predetermined threshold σ0. In this case, the subsequent operations are skipped. Subsequently, at S105 of FIG. 5, as shown by the image B of FIG. 4, image data produced by moving the first image data A by a predetermined length in a palm direction of the hand image is obtained as a second image data B. Specifically, the predetermined length is, for example, 20% to 80% (actual length of about 5 to 20 mm) of the length of the fingertip portion in the direction Y, wherein the fingertip portion is from the first joint of the middle finger. At S106, as shown by the image C of FIG. 4, a non-overlap region, which appears in the fingertip side when superimposing the image data one another, is specified as a tip end region (fingertip region) to in the insertion direction of the hand. The obtained original image data is moved in the lengthwise direction of the palm image and overlapped, and thereby the fingertip region can be easily specified as a non-overlap region. Even when two or more fingers are close to each other and tightly in contact with each other, the fingertip regions, which are in round shapes, can be clearly and separately specified.
Subsequently, at S108, a fingertip determination operation is performed. Specifically, it is determined whether each of the separated and specified tip end regions is a true fingertip region. As shown in FIG. 7, the true fingertip region is determined on the basis of determination whether the size (width) L of the specified tip end region to in the finger width direction is in a stipulated range between an upper limit Wth1 and a lower limit Wth2. The stipulated range is predetermined on the basis of the width of a finger of a common adult. Referring to FIGS. 1, 2A, an occupant inserts a palm beyond a rear periphery of the imaging area 102b toward the input operation surface 102a of the touch panel 12a, i.e., the imaging area 102b of hand imaging camera 12b, while looking at the monitor device (display device) 15 located in the instrument panel. The input operation surface 102a of the center console C is located on the rear side of the monitor device 15 in the lengthwise direction of the vehicle. The input operation surface 102a is located on the front side of a seated occupant in the lengthwise direction of the vehicle. The imaging area 102b of the hand imaging camera 12b is for obtaining an image of a hand of the occupant, which is located on the operation input surface 102a, from the lower side. Therefore, the insertion direction of the hand is assumed to be in parallel with the Y-direction, which is perpendicular to the long side of the imaging area 102b in a horizontally long rectangular shape. The finger width direction is assumed to be in parallel with the X-direction, which coincides with the long side of the imaging area 102b. That is, the finger width direction is assumed to be perpendicular to the insertion direction of the hand when inserted toward the operation input surface 102a. In short, the width of the tip end region ta is fixedly measured in the X-direction, i.e., in the direction of the long side of the imaging area 102b.
FIG. 6 is a flow chart showing a detailed one example of the fingertip determination operation. At S1001, the width W of each of the tip end regions ta, which are separated and specified, is obtained. Specifically, the width W of each of the tip end regions ta can be calculated by obtaining the maximum value Xmax and the minimum value Xmin of an X coordinate of each pixel constituting the tip end region ta and calculating the formula of W=Xmax−Xmin. At S1002, it is determined whether the width W of the tip end region ta specified in the above-described manner is in the above-described range between the upper limit Wth1 and the lower limit Wth2. The touch panel 12a is located on the center console C. It is noted that when an occupant such as a driver and a passenger is seated on the side of the center console C, the occupant often uses the center console C as a pedestal for a baggage. In the present condition, when such a baggage other than a hand is placed on the operation input surface 102a, i.e., the imaging area 102b of the touch panel 12a, an image of the baggage is photographed, instead of a hand of the occupant. According to the present embodiment, the photographed image is determined based on the width W of the tip end region ta, which is extracted from the difference between the obtained first image and the second image. The second image is obtained when the palm (hand) is moved in the Y-direction. Therefore, when a baggage such as a document or a book is placed on the center console C, the extracted and specified width of the tip end region is significantly greater than the upper limit of the range predetermined on the basis of the width of a finger of a common adult. Therefore, such a baggage can be steadily determined as a non-fingertip region. Alternatively, a cellular phone may be placed on the center console C. In this case, for example, an antenna portion protruding from the cellular phone may be imaged as the first tip end region. It is noted that, the width of such an antenna is significantly less than the width of a finger. Therefore, the width of the imaged first tip end region of such an antenna is much less than the lower limit of the range predetermined on the basis of the width of a finger. Thus, such a cellular phone can be properly determined as the non-fingertip region. At S1003, the number N of the fingertip region is specified. At S1004, the total area of the imaged object is specified.
Alternatively, it is conceived that an object, which is in a shape to have the width W within the predetermined range, may be placed on the input surface 102b of the touch panel 12a. In this case, such an object may be possibly recognized to be a true fingertip region erroneously, even though the object is supposed to be a non-fingertip region, For example, a coin may be placed on the input surface 102b. In this case, since the size of the coin is close to the finger width, the width of the determined tip end region caused by the coin and obtained by the contraction operation may be within the predetermined range. Consequently, in this case, the coin may be erroneously determined to be a fingertip region. Here, the difference between an image of a finger and an image of a coin will be described. A finger has a lengthened imaging region including a portion near a finger base and extending to reach a rear end periphery of the imaging area on the side of the inserted palm. On the contrary, a coin has a circular imaging region isolated from the rear end periphery of the imaging area on the side of the inserted palm. In the case of a coin, a background region (“0” pixel region) is caused in the area between the rear end of the circular region of the coin and the rear end periphery of the imaging area on the side of the inserted palm. Therefore, an index value of S/d may be used for determining whether the obtained image is a finger or another object such as a coin, wherein S is a total area of a photographic image, and d is a total distance of the non-overlap region to the rear end periphery of the imaging area. Thus, the above-described erroneous determination can be effectively avoidable by making a determination in accordance with the index value S/d. Specifically, in the case of an object such as a coin, the background region exists on the side of the rear end periphery of the imaging area. Therefore, the total area S is small. Therefore, when the index value S/d is less than the lower limit of the predetermined range, the imaged object may be determined to be a non-fingertip region and may be excluded from the fingertip region. Alternatively, the finger width may be estimated on the basis of the index value S/d. In this case, when the estimated finger width is less than the lower limit of the predetermined range, the imaged object may be determined to be a non-fingertip region and may be excluded from the fingertip region. In the flow chart of FIG. 6, the above-described determination is made at S1005 and S1006.
It is noted that, a finger pad of a finger actually makes contact with the input surface 102b of the touch panel. That is, a middle portion of a finger at a lower side of the fingertip in the Y-direction actually makes contact with the input surface 102b. Therefore, in the image F of FIG. 4, the centroid position calculated in the image E is moved, i.e., offset to the lower side in the Y-direction by a predetermined length, and the offset position is determined as an image fingertip point G. Alternatively, the centroid position determined in the image E may be used as the image fingertip point G. In this case, the operation shown by the image F may be omitted. When the hand is in a specific physical relationship relative to the imaging area 102b, the representative point determined by the algorithm according to the difference between images may not coincide with the true fingertip point. Specifically, when a fingertip portion is out of an imaging area, the determined representative point may not coincide with the true fingertip point. Here, it is supposed that the correspondence coordinate range of the imaging area 102b coincides with those of the input surface 102b of the touch panel and the screen of the monitor device 15. In this case, a true fingertip position may be within the periphery of the imaging area (input surface) 102b, i.e., the screen of the monitor device 15. Alternatively, a true fingertip portion may be out of the imaging area 102b. In this case, the image of the fingertip portion breaks off, i.e., only an image of a middle part of the finger is obtained. In any of the above cases, the tip end region specified according to the difference between images is within the periphery of the imaging area 102b. Even when the fingertip portion is out of the imaging area 102b, an image of a part of a finger is obtained. In this case, the width of the obtained image is possibility within the predetermined range. Therefore, the tip end region obtained in the periphery of the imaging area 102b may be erroneously recognized as a true fingertip region.
In view of the present problem, in the present embodiment, a peripheral region of an effective coordinate range of the imaging area 102b is assigned as an out-of-indication imaging region. An indication area corresponding to the out-of-indication imaging region is out of the effective coordinate range of the screen of the monitor device 15. A tip end of a finger portion, which extends to the outside of the screen, causes an image object region in the out-of-indication imaging region. Therefore, the tip end region specified based on the difference between images and the imaged fingertip point specified by the representative point of the tip end region are caused in the out-of-indication imaging region. On the other hand, as shown in FIG. 13B, when a true fingertip does not enter the out-of-indication imaging region and remains in the periphery of the screen, the tip end region and the imaged fingertip point are caused in the screen. Therefore, when the tip end region ta exists in the out-of-indication imaging region 102e, the tip end region ta is determined to be invalid, and the tip end region ta is not recognized as a true fingertip. Alternatively, when the tip end region ta does not exist in the out-of-indication imaging region 102e, the tip end region ta is determined to be effective, and the tip end region ta is recognized as a true fingertip. For example, as shown in FIG. 14, when multiple imaged fingertip points tp are specified, it is determined whether the multiple imaged fingertip points tp exists in the out-of-indication imaging region 102e one by one. Thus, the determination whether the imaged fingertip point tp is effective or invalid is individually performed for each of the multiple imaged fingertip points tp. In the flow chart of FIG. 6, the above-described determination is made at S1008 to S1010. The rear end of the display screen on an insertion side, from which a hand is inserted in the Y-direction, may coincide with the rear end of the imaging area 102b. In this case, the out-of-indication imaging region 102e may not be assigned to the rear end of the display screen in the Y-direction.
The algorithm for determining whether the tip end region ta is a true fingertip region may employ various other methods. For example, the length of the fingertip portion in the direction Y in the first image for obtaining the second image may be set to a smaller value than a width of a finger of a common adult. In this case, the tip end region ta obtained as the difference between the first image and the second image is apt to be a horizontally long region. Specifically, the X-direction size WX (width) of the horizontally long region is apt to be greater than the Y-direction size WY (length) of the horizontally long region. In the present case, it can be determined whether the tip end region ta is a true fingertip region based on determination whether an X/Y aspect ratio φ (≡WX/WY) of the tip end region ta is in a predetermined range. For example, when a paper or a document is placed on the input surface 102b, the aspect ratio φ becomes extraordinary large. Alternatively, when an antenna of a cellular phone, which is thinner than a finger, is placed on the input surface 102b, the aspect ratio φ becomes a smaller value by a shortage of the X-direction size WX (width). In either case, such a paper or a document and an antenna of a cellular phone can be excluded from the non-fingertip region.
As follows, various embodiments of a user interface engine 103e will be described. As shown in FIG. 3, the operation ECU 10 obtains vehicle speed information; which is detected by the vehicle speed sensor 301, from the body ECU 300 via the serial communication bus 30. The operation ECU 10 determines the present state to be a low-speed traveling state or a stopping state when the obtained vehicle speed information indicates that the vehicle speed is less than or equal to a threshold such as 1 to 10 km/hr (5 km/hr as one example). Alternatively, the operation ECU 10 determines the present state to be a high-speed traveling state (traveling state) when the obtained vehicle speed information indicates that the vehicle speed is greater than the threshold. The user interface engine 103e has interface specification data including low-speed traveling interface specification data (stopping interface specification data) used when the vehicle stops and high-speed traveling interface specification data (traveling interface specification data) used when the vehicle is traveling. When the vehicle stops, the interface specification data is switched to the stopping interface specification data. When the vehicle is traveling, the interface specification data is switched to the traveling interface specification data.
In the stopping interface specification, the operation regions are individual character input soft button SB constituting a character input keyboard KB. The image fingertip point is positioned at a soft button of a preferred character, and a touch operation is performed in a position corresponding to the preferred character on the touch panel 12a. That is, the touch operation is performed to cause an inputted touch point in a region of the soft button. Thereby, a corresponding character is inputted, and a code corresponding to the inputted character is outputted as operation instruction information to the car navigation system.
In the stopping interface specification, the operation region includes individual menu buttons SB1. Similarly to the first embodiment, the image fingertip point is positioned at a soft button of a preferred menu, and a touch operation is performed in a position corresponding to the preferred menu on the touch panel 12a. That is, the touch operation is performed to cause an inputted touch point in a region of the soft button. Thereby, a corresponding menu is selected, and the content of the selected menu is outputted and transmitted as operation instruction information to the car navigation system.
On the other hand, in the traveling interface specification, the number of the menu buttons SB1 is reduced, and a sub-menu button SB2 is indicated. The sub-menu buttons SB2 are at a lower rank than a specific menu in a hierarchical menu. Each of the sub-menu button SB2 is formed and indicated to be rectangular and larger in the horizontal direction than the menu button SB1 in the stopping interface specification. In FIG. 12B, the sub-menu buttons SB2 is a potential destination as one example, and the specific menu is a destination history as one example. Similarly to the first embodiment, the image fingertip point is positioned at a soft button of a preferred menu, and a touch operation is performed in a position corresponding to the preferred menu on the touch panel 12a. That is, the touch operation is performed to cause an inputted touch point in a region of the soft button. Thereby, a corresponding menu is selected, and the content of the selected menu is outputted and transmitted as operation instruction information to the car navigation system.
In the stopping interface specification, when the number of the detected image fingertip point G is one (not shown), a play and stop button P/S is indicated as the operation region in a position corresponding to the image fingertip point G. When the number of the detected image fingertip points G is two, as shown in FIG. 13B, a forward button FF and a rewind button RF are indicated in the positions corresponding to the image fingertip points G. Further, when the number of the detected image fingertip points G is three, as shown in FIG. 13A, the forward button FF, the play and stop button P/S, and the rewind button RF are indicated in the positions corresponding to the image fingertip points G. In either case, a touch operation is performed on a button indicated on the touch panel 12a by at least one finger to select a function corresponding to the touched button. Thus, a content of the selected function is outputted as operation instruction information. In this case, the maximum formation and indication classification number of the operation regions is three when the number of the detected image fingertip points G is three.
In the present screen, a drug operation on the input surface 102b of the touch input device 12 is detected. In the present example, the drug operation is detected according to a temporal change in the inputted touch point TP. It is noted that the drug operation may be detected according to a change in the imaged fingertip point. When the drug operation is performed in the arrangement direction of the album window groups AW, AW′, the title image of the present album window AW is successively switched to the title image of the album corresponding to the reserve album window group AW′, and the switched title image is indicated. Simultaneously, the album name indicated on the album name window ATW is also successively switched. When the direction of the drug operation changes, the switching direction of the albums also changes.
FIG. 17 show a second embodiment of interface specification data. The interface specification data is applied to a menu screen for setting a destination in a car navigation system. The operation regions are individual menu buttons SBS, SBL. The image fingertip point is positioned at a soft button of a preferred menu, and a touch operation is performed on a position corresponding to the preferred menu on the touch panel 12a. Thereby, a corresponding menu is selected, and the content of the selected menu is outputted and transmitted as operation instruction information to the car navigation system. The left drawing of FIG. 17 shows an interface when multiple imaged fingertip points are simultaneously detected. The right drawing of FIG. 17B shows an interface when only one imaged fingertip point is detected. The formation and indication number of the menu buttons SBS when multiple imaged fingertip points are simultaneously detected is set to be greater than the formation and indication number of the menu buttons SBS when only one imaged fingertip point is detected. In the case of the right drawing of FIG. 17, as the formation and indication number of the menu buttons SB1 decreases, the formed size of each menu button SB1 is increased.
FIGS. 18 to 20 show a first example of interface specification data according to a third embodiment. When the imaged fingertip point is detected, the interface specification of the user interface engine 103e is changed according to determination whether a touch operation to the touch input device 12 is simultaneously detected. The interface specification is applied to a map display screen of a car navigation system. In the map display screen, a map indication region MA and a scroll button SB are formed. As shown in FIG. 18, a touch operation is performed to the scroll button SB, and thereby a scroll operation of the map in map indication region MA can be performed.
the alteration unit is further configured to: alter an indicated content on the operation panel according to the traveling state; and alter the instruction output correspondingly to the indicated content.
the alteration unit is further configured to: set a first number of the plurality of operation regions being effective in the high-speed traveling interface specification; set a second number of the plurality of operation regions being effective in the low-speed traveling interface specification; and set the first number to be less than the second number.
the alteration unit is further configured to: set a first size of the plurality of operation regions being effective in the high-speed traveling interface specification; set a second size of the plurality of operation regions being effective in the low-speed traveling interface specification; and set the first size to be larger than the second size.
the alteration unit is further configured to set a first maximum number of the plurality of operation regions in the high-speed traveling interface specification; set a second maximum number of the plurality of operation regions in the low-speed traveling interface specification; and set the first maximum number to be less than the second maximum number.
the second indication control unit is further configured to: detect a drug operation along the operation surface of the input device; switch a plurality of operation regions, which respectively have different instruction outputs, to be effective; and indicate switched one of the plurality of operation regions on the screen, and
the alteration unit is further configured to cause the second indication control unit to: continually switch the plurality of operation regions during one continuous drug operation in the low-speed traveling interface specification; and intermittently switch the plurality of operation regions each time one drug operation arises in the high-speed traveling interface specification.
the user interface engine includes an output unit configured to: output a first instruction output corresponding to an operation pattern on detection of the operation pattern; and output a second instruction output associated with an operation region on selection of the operation region by the detected touch point or the detected fingertip, and
the alteration unit is further configured to cause the output unit to: activate at least the second instruction output according to selection of the operation region in the low-speed traveling interface specification; and activate only the first instruction output according to detection of the operation pattern in the high-speed traveling interface specification.
the alteration unit is further configured to: set a first number of the plurality of operation regions being effective when the fingertip detection unit detects a plurality of fingertips: set a second number of an effective operation region when the fingertip detection unit detects one fingertip; and set the first number to be greater than the second number.
the second indication control unit is further configured to: detect a drug operation along the operation surface of the input device; and scroll an image being indicated on the screen in a direction corresponding to the drug operation, and
the alteration unit is further configured to: set a first scroll speed of the image when the input device detects a touch operation simultaneously when the fingertip detection unit detects a fingertip; set a second scroll speed when the input device does not detect a touch operation; and set the first scroll speed to be lower than the second scroll speed.
the second indication control unit is further configured to: detect a drug operation along the operation surface of the input device; and scroll a hierarchical menu being indicated on the screen, and
Publication number: 20100277438
Patent Grant number: 8593417
Inventors: Takeshi Kawashima (Nisshin-city), Koichi Masuda (Obu-city)
Application Number: 12/799,646