Pointing input system and method using one or more array sensors

In a pointing input system and method, a pointer applies a light spot on a screen, an array sensor shoots on the screen to generate a first data for an identification system to retrieve a second data therefrom, the second data includes the position information of the light spot for an information system to apply a correlated output on the screen. The second data is generated based on an optical distortion information and a spatial rotation and displacement information determined by an alignment procedure that comprises applying an input for alignment on the screen for the array sensor to shoot to generate an alignment data, and comparing the alignment data with a reference data.

FIELD OF THE INVENTION

The present invention is related generally to a system and method for direct inputs on a screen and more particularly, to a pointing input system and method for an information system.

BACKGROUND OF THE INVENTION

Current input apparatus available for information systems includes keyboard, mouse, trackball, light pen, and touch panel. The trackball is not suitable to be used for writing on an upright screen. The light pen is only available for the inputs on a scanning screen such as Cathode-Ray Tube (CRT) screen, but not for the inputs on a high resolution screen. The touch panel is disadvantageous to alignment and portability for larger-scale screens. Due to the significant barrel or pincushion distortion of the optical lens, the input systems using conventional sensors may often suffer the alignment degradation caused by the mistakes or unintentional touches to the equipment, and even have to be suspended during its use accordingly. Therefore, single keyboard and/or single mouse installed on a computer system is still relied on currently for the inputs to an information system using an upright large-scale screen. However, single input apparatus is inconvenient for use in the situation where multiple inputs from several persons or opinion exchanges between several persons are required.

Therefore, it is desired a pointing input system and method easy for alignment, capable of precise positioning, and available for high resolution applications.

SUMMARY OF THE INVENTION

One object of the present invention is to provide a pointing input system and method for an information system.

Another object of the present invention is to provide a pointing input system and method for direct inputs on an upright large-scale screen.

In a pointing input method according to the present invention, a light spot is applied on a screen, the screen is shot by an array sensor to generate a first data, a second data including the position information of the light spot is retrieved from the first data, and an output corresponding to the light spot is applied on the screen. The second data is generated from the first data based on an optical distortion information and a spatial rotation and displacement information determined by an alignment procedure that comprises applying an input for alignment on the screen for the array sensor to shoot on the screen to generate an alignment data, and comparing the alignment data with a reference data.

In a pointing input system according to the present invention, a first buffer stores a reference data and an alignment data generated in an alignment procedure, a processor analyzes the alignment data and compares the alignment data with the reference data to obtain an optical distortion information and a spatial rotation and displacement information of an optical lens in an array sensor that is used to shoot on a screen in the alignment procedure, a second buffer stores the optical distortion information and spatial rotation and displacement information, a pointer applies a light spot on the screen for the array sensor to shoot to generate a first data, an identification system retrieves a second data including the position information of the light spot from the first data, and a display system applies an output corresponding to the light spot on the screen.

DETAILED DESCRIPTION OF THE INVENTION

Transformation Between Two Coordinate Systems

In a space10, as shown inFIG. 1, an array sensor12has a focus plane14apart from the array sensor12with a distance L, a point M(X, Y, Z) in the space10is mapped to a point m(x, y, z)=m(X×L/Z, Y×L/Z, L) on the focus plane14, where X, Y, Z, x, y, and z are the coordinates of the respective points M and m. If an optical lens16for mapping onto the array sensor12has an optical distortion, due to the polar symmetric to the central point of the optical lens16, the spatial relationship between the optical lens16and the original point of the array sensor12is first determined, and further to transform the original coordinates to a polar coordinates as in the following

where matrix A is the transformation matrix of the combination of the array sensor12and optical lens16, fxand fyare the focus distances along the X axis and Y axis, cxand cyare the coordinates of the central point of the image mapped by the optical lens16onto the array sensor12, x and y are the coordinates at the array sensor12with the point (cx, cy) as the central point, matrixes R and T are the transformation matrix resulted from the rotation and displacement in the space10, k1is the second-order radial distortion, k2is the fourth-order radial distortion, p1is the second-order tangent distortion, and p2is the fourth-order tangent distortion.

First Embodiment of Alignment Procedure

FIG. 2shows an alignment procedure20afor a pointing input system applied for an information system using front projection. To identify the shooting range of an array sensor22aon a screen21a, it is used a viewing window25aon the array sensor22ato inspect on the screen21aor light beams emitted by high intensity Light-Emitting Diodes (LEDs) or laser diodes24aon the array sensor22ato project light spots on the screen21a. Then an input for alignment is applied on the screen21a, for example displaying a test pattern on the screen21aby a front projector23a.FIG. 3shows three exemplary test patterns, chessboard pattern30, square pattern32, and cross pattern34. However, the test pattern may be other predetermined patterns in some other embodiments. The array sensor22ashoots the test pattern on the screen21ato generate an alignment data26athat is further sent to a computer system27a. The alignment data26aincludes the image information of the test pattern generated by the array sensor22a, and an image recognizing software and spatial displacement and rotation calculation tool programs28aare running on the computer system27ato analyze the relative positions and three-dimensional relationship of several points on the test pattern from the alignment data26aand a reference data. Briefly, the alignment data26ais the one obtained by shifting and rotating the original test pattern in a three-dimensional space, and by comparing the alignment data26awith the reference data, the spatial relationship between the screen21aand array sensor22ais determined. As a result, a spatial rotation and displacement information is obtained, which describes the spatial transformation between the screen21aand array sensor22a. By comparing the relative positions between the several points, in association with the lens distortion equation such as EQ-4, it is determined an optical distortion information for the optical lens in the array sensor22ato further modify the spatial relationship between the screen21aand array sensor22a. The screen21amay have a planar surface, a regularly curved surface, or an irregularly curved surface. In the circumstances of regularly or irregularly curved surface such that the arithmetic computation for the spatial transformation is too complicated, bilinear interpolation may be used to calculate the coordinates of the several points in the alignment data26aand reference data to conduct the spatial transformation relationship. The information s generated by the alignment procedure20apictures the spatial relationship between the screen21aand array sensor22aand the optical distortion of the optical lens in the array sensor22a.

Second Embodiment of Alignment Procedure

FIG. 4shows an alignment procedure20bfor a pointing input system applied for an information system using rear projection. To identify the shooting range of an array sensor22bon a screen21b, it is used a viewing window25bon the array sensor22bto inspect on the screen21bor light beams emitted by high intensity LEDs or laser diodes26bon the array sensor22bto project light spots on the screen21b. Then an input for alignment is applied on the screen21b, for example applying light spots at several specific positions on the screen21aby a pointer24b. The light spot is reflected to the array sensor22bby a mirror27b. The intensive light from a rear projector23bto project on the screen21bmay be reflected by the mirror27bto generate highly bright spots on the array sensor22b, causing the identification more difficult. Therefore, an optical filter28bis arranged in front of the array sensor22bto filter out the intensive light from the projector23b, and thereby to specify the light source of the pointer24bfrom that of the projector23b. The light spots for alignment are shot by the array sensor22bto generate an alignment data29bincluding the image information of the light spots on the screen21b. The alignment data29bis sent to a computer system30b, where an image recognizing software and spatial displacement and rotation calculation tool programs31bare running to analyze the relative positions and three-dimensional relationship of several points in the alignment data29band a reference data. By comparing with the reference data that includes the spatial information regarding to the specific positions on the screen21bto be applied with the light spots, a spatial rotation and displacement information is obtained to picture the spatial transformation between the screen21band array sensor22b. The screen21bmay have a planar surface, a regularly curved surface, or an irregularly curved surface. In the circumstances of regularly or irregularly curved surface such that the arithmetic computation for the spatial transformation is too complicated, bilinear interpolation may be used to calculate the coordinates of the several points in the alignment data26aand reference data to conduct the spatial transformation relationship. The information generated by the alignment procedure20bpictures the spatial relationship between the screen21band array sensor22b.

First Embodiment of Pointing Input System

FIG. 5shows a pointing input system40applied for an information system using single front projector41and single array sensor42. In the pointing input system40, after an alignment procedure20aillustrated inFIG. 2, a pointer46could be used for direct inputs on a screen43. The light spot47projected by the pointer42on the screen43is shot by the array sensor42to generate a first data including the image information of the light spot47sent to a computer system44, where an identification system49running on the computer system44retrieves a second data including the position information of the light spot47from the first data based on the optical distortion information and spatial rotation and displacement information that are obtained by the alignment procedure20a. The second data is provided for the projector41to display a correlated output on the screen43. InFIG. 5, the correlated output is an image of the light spot47, while in other embodiments, it may be a cursor moved to the position of the light spot47, or another output generated in response to a command from the pointer46. In the system40, an optical lens45and an optical filter48are arranged in front of the array sensor42for mapping onto the array sensor42and filtering out the optical noise for the array sensor42, thereby enhancing the identification carried out by the computer system44.

Second Embodiment of Pointing Input System

FIG. 6shows a pointing input system50applied for an information system using single front projector51and multiple array sensors52. After an alignment procedure20aillustrated inFIG. 2, the system50allows a pointer46to directly input on a screen53. The array sensors52are arranged in different directions to shoot on the screen53, and thus, if one of them is unable to shoot the light spot on the screen53properly, another is switched instead to shoot on the screen53, so as to achieve the purpose of full viewing angle without any loss.

Since several array sensors52are provided in the system50, the screen53may have very large area for display. The other operations are referred to the first embodiment40illustrated inFIG. 5. The identification system is not shown inFIG. 6for simplicity.

Third Embodiment of Pointing Input System

FIG. 7shows a pointing input system60applied for an information system using multiple front projectors61, multiple array sensors62, and multiple pointers63. In this system60, a screen64is defined to have several projection regions each one is responsible by a projector61to display thereon and by an array sensor62to shoot thereon, so as to improve the resolution. After an alignment procedure20aillustrated inFIG. 2, the pointers63are allowed in this system60for direct inputs on the screen64. By using the pointers63, it is achieved multiple inputs from several persons or opinion exchanges between several persons, without any additional equipments. Using the pointers63for direct inputs on the screen63may be referred to the first embodiment40illustrated inFIG. 5. The identification system is not shown inFIG. 7for simplicity.

Fourth Embodiment of Pointing Input System

FIG. 8shows a pointing input system70applied for an information system using rear projection. After an alignment procedure20billustrated inFIG. 4, it is allowed direct inputs on a screen73by a pointer75in this system70. With a mirror74, a rear projector71projects an image reflected on the screen73. The mirror74also assists an array sensor72in shooting on the screen73. The light spot76projected by the pointer75on the screen is reflected by the mirror74to the array sensor72to generate a first data including the image information of the light spot76. The first data is sent to a computer system (not shown) for an identification system to retrieve a second data including the position information of the light spot76from the first data based on the spatial rotation and displacement information that are obtained in the alignment procedure20b. The second data is provided for the projector71to display a correlated output on the screen73. InFIG. 8, the correlated output is an image of the light spot76, while in other embodiments, it may be a cursor moved to the position of the light spot76, or another output generated in response to a command from the pointer75. For the intensive light from the projector71to project on the screen73may be reflected by the mirror74to generate highly bright spots on the array sensor72, causing the identification more difficult, an optical filter78is arranged in front of the array sensor72to filter out the intensive light from the projector71, and thereby to specify the light source of the pointer75from that of the projector71, improving the identification and repeatability.

Hardware of Pointing Input System

FIG. 9shows a pointing input system80according to the present invention, in which a display system81is provided to display on a screen82, an array sensor83is prepared to shoot on the screen82, and a pointer87is used to apply light spots on the screen82either in an alignment procedure or for a pointing input. The array sensor83may comprises Charge-Coupled Detector (CCD) or Complementary Metal-Oxide-Semiconductor (CMOS) sensor. In an alignment procedure, a buffer84is used to store a predetermined reference data that includes the spatial information of a test pattern to be displayed on the screen82or several specific positions on the screen82to be applied with light spots thereon. Particularly, the spatial information includes the relative positions of several points in the test pattern or the several specific positions on the screen82, for example the relative directions and distances therebetween. The buffer84also stores an alignment data in an alignment procedure. The alignment data is generated by shooting the test pattern or light spots at the several specific positions on the screen82by the array sensor83, and therefore includes the image information of the shot test pattern or light spots. A processor85is provided to identify and analyze the alignment data, for example with image recognizing software and displacement and rotation calculation tool programs, in comparison with the reference data to evaluate the spatial relationship between the array sensor83and screen82and the optical distortion of the optical lens in the array sensor83. In this process, for example, the processor85analyzes the relative positions of several points in the test pattern or at the several specific positions on the screen82between the alignment data and reference data and their three-dimensional spatial relationship, to obtain a spatial rotation and displacement information. The processor85further compares the three-dimensional spatial coordinates of the several points to determine an optical distortion information of the optical lens in the array sensor83based on an optical lens distortion equation such as EQ-4. The optical distortion information and spatial rotation and displacement information are stored in a buffer86for use in a pointing input. For a pointing input, a light spot is applied on the screen82by the pointer87, and the array sensor83shoots on the screen82to generate a first data including the image information of the light spot. The first data is sent to an identification system88, where it is identified for the light spot and a second data including the position information of the light spot is generated based on the optical distortion information and spatial rotation and displacement information provided by the buffer86. The second data is provided for the display system81to apply a correlated output on the screen82. In this embodiment, the identification system88comprises a buffer88ato store the first data generated by the array sensor83, a register88cto provide an optical feature condition, and a processor88bto check the light spot from the first data to find out the nearby pixels satisfying the optical feature condition to define an output spot, and to calculate the information including the size, average brightness, hue, length, width and area of the light spot that may be also included in the second data.

The screen82may have a planar surface, a regularly curved surface, or an irregularly curved surface, and may be an active screen such as CRT screen, LCD screen, plasma screen, and rear projection screen, or a passive screen such as scattering screen of front projector. All the data to be used or processed may be stored and processed in a same computer system or separately stored and processed in different computer systems.

Flowchart of Alignment Procedure

As shown inFIG. 10, in a flowchart90of an alignment procedure for a pointing input system according to the present invention, the shooting range of an array sensor on a screen is first identified in step91, an input for alignment is applied on the screen in step92, the screen is shot by the array sensor in step93, an alignment data is generated in step94, the alignment data is identified and compared with a reference data in step95, and an optical distortion information and a spatial rotation and displacement information is finally determined in step96. The alignment data includes the image information of the input for alignment shot by the array sensor, and the reference data includes the predetermined image information of the input for alignment that is provided to apply on the screen, and therefore from the alignment data and reference data, the spatial relationship between the array sensor and screen and the optical distortion of the lens in the array sensor are able to be determined. Specifically, the optical distortion information and spatial rotation and displacement information picture the optical distortion of the optical lens in the array sensor and the spatial relationship between the array sensor and screen, and with which, the light spots applied on the screen after the alignment procedure are precisely identified for their positions.

Flowchart of Pointing Input Method

FIG. 11shows a flowchart100of a pointing input method according to the present invention. After the step90for alignment as shown inFIG. 10is completed, in step101apointer is used to apply a light spot on a screen, the screen is shot by an array sensor in step102, and a first data including the image information of the shot light spot is generated in step103. The first data is then identified in step104to generate a second data including the position information of the light spot on the screen based on the optical distortion information and spatial rotation and displacement information determined by the alignment procedure90. The second data is provided for an information system to generate a correlated output applied on the screen by a display system in step105.

Embodiment of Pointer

As shown inFIG. 12, a pointer110available for a pointing input system according to the present invention comprises a set of light sources112having different properties, several buttons113, a roller114and a switch115. The light sources112are used for applying light spots on a screen, and the buttons113and roller114are used to switch the light sources112between different optical features, such as brightness, hue, size and shape. The optical features of the light sources112may be used for a trigger signal of an action, for example a pressing of a left button of a mouse. The light sources112may output light beams having one of several specific wavelengths or blinking with one of specific patterns, to be distinguished from the environment light and the light emitted by the display system, and to be sensitive by the array sensor, and it may also serve as an indication signal. Infrared Ray (IR) LED, visible light LED, and laser diode may be used for the light sources112. The switch115may be a contacting or non-contacting switch, to determine the pointer110to touch on the screen, so as to produce different input effects, for example to be at an input state. If the switch115is able to generate an analog output, it may be used to determine a signal representative of the input pressure, so as to feature a pen stroke. The pointer110may be used as a pen for direct inputs on a screen, for example applying a light spot on a screen by pressing the switch115, or drawing a line by continuously moving a light spot on a screen. The pointer110may also be used as a mouse for one click, double click, or drag on a screen.

Identification of Light Spot

FIG. 13shows an illustrative diagram for an identification of a light spot according to the present invention. The image of a light spot mapped onto an array sensor generally occupies a size of tens to hundreds of pixels, but not a single pixel, and therefore the image of the light spot captured by the array sensor will have a distribution of optical features such as the brightness. Based on a background noise121and a threshold122to determine the input spot, for example by selecting the position123having the maximum of the optical feature, or the position124having the center mass of the optical feature, or the central position125in the range having the optical feature close to the threshold122as the position of the light spot, it is able to eliminate the interference of the background noise, and high stability and high precision are obtained for the light spot.

After identifying the position of a light spot, the position is stored to further monitor the continuous movement of the light spot and track the relative position of the light spot. Preferably, the relative position of the light spot is used for an information of determining a trigger signal representative of an action or a pen stroke.

While the present invention has been described in conjunction with preferred embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the spirit and scope thereof as set fourth in the appended claims.