Pen input device and method for tracking pen position

The pen input device has both a movement detection system and a position detection system that collectively track the position of the pen input device during a writing operation on a writing surface. The pen input device has a writing tip capable of performing a writing operation in contact with the writing surface. A controller is operable to activate the movement detection system continuously during the writing operation and to activate the position detection system intermittently during the writing operation. The movement detection system is operable to determine movement data that represent changes between successive positions of the writing tip. The position detection system is operable to determine position data that represents a position of the writing tip on the writing surface.

BACKGROUND OF THE INVENTION

Recently, pen input devices have become available that electronically capture pen strokes on a surface (hereinafter referred to as “writing operations”) by sensing the time-dependent position of the pen on the surface and converting the pen positions to writing operations. However, traditional incremental movement detection techniques, such as those used in optical mice, fail when the pen input device is lifted off of the surface. Therefore, position detection techniques have been developed to determine the actual location (position) of the pen input device on the surface.

One position detection technique uses paper pre-treated with optically recognizable fiducials that allow the pen input device to determine its position relative to the fiducials. For example, as described in U.S. Pat. No. 6,650,320, a laser mounted on a digital pen input device directs coherent light towards special bar-coded paper. The reflected interference patterns are imaged and processed to determine the position of the pen input device relative to the known bar code patterns. However, requiring special paper increases the cost and limits the usage of such pen input devices.

Another position detection technique uses passive optical fiducials to determine the position of the pen input device. Typically, the pen input device is provided with wide field of view optics (e.g., panoramic optics) that enable an image of the surroundings of the pen input device to be taken. As long as a sufficient portion of the paper and its surroundings (e.g., three edges of the paper) are visible in the image, the position of the pen input device, along with the orientation of the pen input, can be computed. As writing operations are performed, the information written on the paper during the writing operations serve as additional fiducials.

However, the image processing involved in determining the position of the pen input device using passive optical fiducials is complex and computationally intensive. Additionally, frame rates in excess of 100 Hz are required to track the movement of the pen input device. In addition, some of the movement of the pen input device can be attributed to involuntary movements of the user's hand that result in no significant movement of the writing tip of the pen input device. Performing complex computations merely to determine that no significant movement occurred at the writing tip of the pen is inefficient and undesirable, especially for wireless, battery-powered pen input devices.

Some position detection techniques use the edges of the paper as references with respect to which the position of the pen input device is determined. However, under some circumstances, there may not be enough optical contrast between a blank sheet of paper and the background to enable the position of the pen input device to be reliably determined.

Therefore, what is needed is a technique for determining the path of a pen input device on a writing surface with reduced computational complexity and that does not require special paper.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a pen input device including both a movement detection system and a position detection system that collectively track the position of the pen input device during a writing operation on a writing surface. The pen input device has a writing tip capable of performing a writing operation in contact with the writing surface. A controller is operable to activate the movement detection system continuously during the writing operation and to activate the position detection system intermittently during the writing operation. The movement detection system is operable to determine movement data that represent changes between successive positions of the writing tip. The position detection system is operable to determine position data that represents a position of the writing tip on the writing surface.

In one embodiment, the movement detection system includes a movement optical image sensor and transfer optics. The movement optical image sensor is for capturing successive movement images in response to light returned from the writing surface and for producing movement image data representing the successive movement images. The image transfer optics transfer the light from the writing surface to the movement optical image sensor. The position detection system includes a position optical image sensor and image transfer optics. The position optical image sensor is for capturing a position image in response to light returned from the writing surface and for producing position image data representing the position image. The image transfer optics transfer the light from the writing surface to the position optical image sensor.

Non-optical movement detection systems are also possible.

In another aspect, the invention provides a method for capturing the position of the writing tip of a pen input device on a writing surface during a writing operation. The method comprises determining movement data while the writing tip contacts the writing surface during the writing operation; determining position data at least once during the writing operation; and linking the position data with the movement data. The movement data represents changes between successive positions of the writing tip on the writing surface. The position data represents a position of the writing tip on the writing surface.

In a final aspect, the invention provides a method for determining the position of the writing tip of a pen input device on a writing surface during writing operations. The method comprises determining movement data representing changes between successive positions of the writing tip during at least two of the writing operations and determining position data representing a position of the writing tip on the writing surface using marks made on the writing surface by the at least two of the writing operations as positional references.

Considerable savings in computation complexity and power consumption are achieved by using movement detection instead of position detection during each writing operation. In addition, the precision with which the position detection system need operate are reduced, resulting in further reductions in cost and complexity.

DETAILED DESCRIPTION

Activities that will be referred to in this disclosure as writing and that encompass writing, printing, drawing, etc., involve a user making strokes with a pen input device on a writing surface. During each stroke, the pen input device contacts the writing surface and typically makes a mark thereon. The process of making a stroke during the activity of writing will be called a writing operation. Additionally, at the end of each stroke, the user performs a repositioning operation in which the user moves the pen input device so that it no longer contacts the writing surface and, with the pen input device out of contact with the writing surface, re-positions the pen input device to the beginning of the next stroke. The user then moves the pen input device into contact with the paper to make the next stroke.

Most systems of writing are based on characters composed of one or more strokes. Examples of the word character as used in this disclosure are a single letter of the Roman alphabet, a word or part of a word written in the Roman alphabet using cursive writing and one kanji of a kanji-based writing system such as Chinese or Japanese. Determining the relative positions of the strokes constituting a character is desirable to enable the character to be accurately captured for reproduction on a screen or to enable the character to be reliably recognized by character recognition.

FIG. 1illustrates an exemplary pen input device10in accordance with embodiments of the invention. Pen input device10has a writing tip20that a user moves relative to a writing surface30by manipulating the pen input device. Typically, the writing tip is capable of making visible marks on the writing surface30. However, some embodiments do not require the writing tip to make visible or otherwise optically detectable marks on the writing surface.

Pen input device incorporates a proximity sensor60, a movement detection system160and a position detection system170. The proximity sensor determines whether the writing tip20of the pen input device is in contact with the writing surface30. The movement detection system160incorporates a simple, low power movement detector that operates while the proximity detector60indicates that the writing tip20is in contact with the writing surface30to detect the movement of the writing tip20relative to the writing surface30. Hence, the movement detection system160detects the shape of each stroke.

In some embodiments, the movement detector is similar to that of an optical mouse. Other embodiments employ other types of movement detectors.

The position detection system170operates intermittently during each writing operation to make a determination of the position of the writing tip20on the writing surface30. The position detection system170typically operates once at the beginning of each writing operation when the proximity detector60indicates that the user has moved the writing tip20into contact with the writing surface30at the beginning of the stroke to determine the position of the writing tip at the beginning of the writing operation. The position detection system170may additionally operate when the proximity detector60indicates that the user has moved the writing tip20out of contact with the writing surface30at the end of each stroke to determine the position of the writing tip20on the writing surface30at the end of the writing operation. From positions of the writing tip20on the writing surface30determined at the beginning and optionally, the end of each writing operation, the position detection system determines the relative positions of the strokes and the orientations of the strokes on the writing surface30. This allows the strokes to be assembled into characters for display on a screen or for recognition by a handwriting recognition system.

The position detection system170is more complex and consumes more power than the movement detection system160but is only activated intermittently. As noted above, the position detection system is typically activated to make a single position determination at the beginning and end of each writing operation. Consequently, the average power consumption of the position detection is relatively low notwithstanding its greater instantaneous power consumption. A further reduction in power consumption is obtained by activating the movement detection system160only when the proximity detector60indicates that the writing tip20is in contact with the writing surface30.

The writing surface30can be any type of flat surface on which a user of the pen input device10can write (e.g., paper, wood, laminate, painted surface, etc.) and is not limited to any particular type of special surface. In embodiments other than those described below with reference toFIGS. 5-7, in which visible marks made on the writing surface are used as positional references, the writing tip20need not leave a visible mark on the writing surface.

The movement detection system160acquires a series of low resolution, narrow view images of the surface30during each writing operation. An image acquired by the movement detection system160will be called a movement image. In response to the movement images, the movement detection system160produces movement image data202from which changes between successive positions of the writing tip20on the writing surface30are determined during each writing operation. The movement detection system typically detects changes between the position of the writing tip in successive movement images.

The position detection system170typically acquires no more than one high resolution, wide view image of the writing surface30at the beginning of each writing operation and another high resolution, wide view image of the writing surface30at the end of each writing operation. An image acquired by the position detection system170will be called a position image. In response to each position image, the position detection system produces position image data from which a respective position of the writing tip20on the writing surface30is determined.

Thus, the position detection system170is used mainly for determining the positions and orientations of the strokes on the writing surface, while the movement detection system160is used to capture the shape of the stroke made during each writing operation.

As noted above, the proximity sensor60detects whether the writing tip20of the pen input device10is in contact with the writing surface30. It should be noted that the term “in contact with” as used in this disclosure encompasses the writing tip20touching the writing surface30and the writing tip20being within a defined proximity of the writing surface30. The proximity sensor60provides a proximity signal whose state indicates whether the writing tip20is in contact with the writing surface30. In the examples described below, the state of the proximity signal indicating that the writing tip is in contact with the writing surface30will be called the contact state and the other state of the proximity signal will be called the non-contact state. However, the proximity signal is not limited to this example.

The proximity signal is used to activate the movement detection system160and the position detection system170. For example, in one embodiment, a change in the state of the proximity signal from contact to non-contact, indicating that user has moved the writing tip20out of contact with the writing surface30, deactivates the movement detection system160to reduce power consumption and momentarily activates the position detection system170to determine the position of the writing tip20on the writing surface30at the end of the writing operation.

In another embodiment, a change in the state of the proximity signal from non-contact to contact, indicating that the user has moved the writing tip20into contact with the writing surface30, momentarily activates the position detection system170to capture a position image of the writing surface30, and activates the movement detection system160to capture a series of movement images to detect movement of the writing tip20on the writing surface30while the writing tip remains in contact with the writing surface. Movement image data derived from the series of movement images captured by the movement detection system160during the writing operation is used to determine movement data representing changes between successive positions of the writing tip20on the surface30during the writing operation. The movement data is used to determine the magnitude of the movement of the writing tip20in two orthogonal directions (Δx, Δy) between successive ones of the movement images. Thus, the movement data for a particular writing operation includes multiple sets of incremental movement data (Δx, Δy), one set for each movement image acquired during the writing operation. The sets of incremental movement data are concatenated to describe the movement of the writing tip20across the surface30from the original position of the writing tip20at the beginning of the writing operation.

For example, in one embodiment, the movement data for a particular writing operation includes data points each of which quantifies the incremental movement (Δx, Δy; or Δr, Δθ) of the writing tip20relative to the writing surface30between successive movement images.

Position image data produced by the position detection system170is used to determine position data that represents the position of the writing tip20of the pen input device10on the writing surface30at the beginning and the end of each stroke written by a respective writing operation. For example, in an exemplary embodiment, when the proximity signal changes from non-contact to contact, indicating that the user has moved the writing tip20into contact with the writing surface30, the position detection system170is momentarily activated to generate position data defining the position of the writing tip20on the writing surface30at the beginning of the stroke. This change in the proximity signal additionally causes the movement detection system160to acquire a first one of the successive movement images. Thus, when the proximity signal changes state from non-contact to contact, the position detection system170acquires a position image to determine the position of the writing tip20on the writing surface30when the writing tip20initially contacts the writing surface30at the beginning of the stroke. Additionally, during the writing operation that follows the initial contact, the movement detection system160acquires the series of movement images used to determine the movement of the writing tip20relative to the writing surface30while the writing tip20is in contact with the writing surface30writing the stroke. Additionally, when the proximity signal changes state from contact to non-contact at the end of each stroke, the movement detection system160is deactivated, and the position detection system170is momentarily activated to acquire a second position image from which the position detection system determines the position of the writing tip20on the writing surface30at the end of the stroke.

Thus, the position detection system170operates intermittently, typically acquiring and processing a single image at the beginning of each stroke and acquiring and processing a single image at the end of the stroke. Thus, the effective frame rate of the position detection system is greatly reduced compared with a conventional position detection system. The position detection system170need not finish its determination of the position of the writing tip20on the writing surface30before the user begins to write the stroke: the determination processing can continue as the stroke is being written. As a result, the position detection system processes the position image data at a frame rate commensurate with the average frequency with which the state of the writing tip20changes from contact to non-contact. Thus, the time available for the position detection system to perform the relatively complex computational operation of processing the position image data to determine the position of the writing tip20on the writing surface30substantially greater than that available in a conventional position detection system. This results in a significant saving in power.

FIG. 2shows a more detailed view of the pen input device10in accordance with embodiments of the present invention. The pen input device10has an elongate hollow body15. The pen body15includes a tapered portion40extending to the writing tip20at one end of the pen body15. The tapered portion40is translucent to allow light to enter and exit the pen body15. For example, the tapered portion40may be formed of glass, plastic or other material that is transparent to light. The elongate shape of the pen body15is designed to enable a user to manipulate the pen body15to deposit ink30, graphite or other substance onto a writing surface during a writing operation. However, in embodiments that do not use marks made on the writing surface by the writing operation as positional references, as will be described in more detail below, or that otherwise do not rely on such marks, the pen input device10can be a stylus-type device that does not deposit ink or other substance onto the writing surface.

The pen input device10also includes a light source50located in the pen body15for illuminating the writing surface30through the tapered portion40of the pen body15. The light source50can be any suitable source of electromagnetic radiation, e.g., infrared or visible light. By way of example, but not limitation, the light source50can be a single light emitting diode (LED), multiple LEDs arranged to illuminate different portions of the writing surface30or an array of LEDs designed to emit light at a desired average intensity.

The wavelength of light emitted by the light source50is typically selected to maximize the intensity of the light returned from the writing surface30and to enable the light returned from the writing surface to be distinguished from unwanted light signals. In addition, the wavelength of the light can be chosen based on user or manufacturer preferences. For example, some manufacturers may prefer blue light to red light in certain applications. The light source50can be in an “on state” in a continuous mode with either a steady or variable illumination intensity. Alternatively, the light source50may be operated in a duty-cycle mode, where the light source50is pulsed on and off to control the average light intensity returned from the writing surface. The intensity of illumination can be controlled using any known technique.

In embodiments in which the movement detection system160and the position detection system170are sensitive enough to operate using ambient light, the light source50may be omitted. In embodiments in which only the position detection system170needs artificial light, the light source50may be momentarily activated while the position detection system170is acquiring a position image.

Image transfer optics70within the pen input device directs light returned from the writing surface onto arrays of photo detectors that constitute a movement optical image sensor80and a position optical image sensor140. The movement optical image sensor80is part of the movement detection system160, and the position optical image sensor140is part of the position detection system170. The image transfer optics70include one or more optical systems for directing light returned from the writing surface30towards the image sensors80and140. For example, the image transfer optics70can include a narrow-angle lens for directing the returned light towards the movement optical image sensor80and a wide-angle lens for directing the returned light towards the position optical image sensor140. The image sensors80and140can be located in the same region of the pen input device10or in different regions of the pen input device10. For example, the narrow-angle lens and movement optical image sensor80can be located near the writing tip20of the pen input device10, whereas the wide-angle lens and position optical image sensor140can be located on the side or the top of the pen input device10, remote from the writing tip20. In addition, in other embodiments, one or more light-guiding devices, e.g., fiber optics, can be included in the pen input device10to guide one or both of the illumination light from the light source50to the writing surface and the light returned from the writing surface to the optical image sensor80or140.

Both the movement optical image sensor80and the position optical image sensor140can be a CCD (Charge Coupled Device), a CMOS—APS (Complimentary Metal Oxide Semiconductor—Active Pixel Sensor) or any other type of light sensor. Optical image sensors80and140have advantages over thermal or capacitive image sensors due to the magnification/demagnification mechanisms that can be used with optical image sensors to reduce the silicon area. Thermal and capacitive image sensors typically require the silicon area to be equivalent in size to the sensing area. In addition, capacitive image sensors can be susceptible to electrostatic discharge, which can decrease the signal-to-noise ratio, and thus degrade the image.

The movement optical image sensor80acquires a series of movement images of the writing surface, and derives from each movement image movement image data representing the movement image. The movement image data includes the intensity of the returned light measured at each photo detector of the movement optical image sensor80for each movement image. Each photo detector captures one picture element (pixel) of the movement image. Each photo detector has a photo sensitive region, with the spacing between the photo-detectors designed to achieve the desired spatial resolution of the sensor80. The photo detectors can be, for example, photodiodes or phototransistors arranged in an array. The size of the array is dependent upon the magnification of the optics. For example, magnifications less than unity, near unity or above unity can be used, depending on the size constraints of the tapered portion40of the pen body15and the manufacturer's preferences.

The movement optical image sensor80provides movement image data (e.g., raw pixel values) for each of the movement images to a processor100located in the pen body15. The processor100processes the movement image data to determine movement data quantifying the incremental change in the position of the writing tip20on the writing surface30between successive movement images during each writing operation performed by the pen input device10, as will be described in more detail below with reference toFIG. 3. The processor100can include any microprocessor, microcontroller, digital signal processor, programmable logic device or other type of processing device capable of performing the functions described herein. In addition, the processor100can include multiple processors or can be a single general-purpose processor capable of executing a number of algorithms. In addition, in other embodiments, at least a portion of the processor100is incorporated in the same chip as the movement image sensor80.

The movement data representing the changes between successive positions of the writing tip20on the writing surface30during each writing operation can be stored in a memory110for later processing and/or can be output to an external device via I/O unit150. The I/O unit150can provide a direct wired connection to an external device, a networked connection to a remote device or a wireless connection to an external and/or remote device. For example, the movement data can be transferred to a handwriting recognition application executing on a personal computer for converting the movement data to alpha-numeric characters and graphical representations.

The memory110is a memory device, such as random access memory (RAM), flash memory, or any other type of storage device. Additional processing information (not shown) can also be stored on the memory110and accessed by the processor100. For example, such processing information can include various processing parameters, such as algorithms that can be used to process the movement image data and determine the movement data from the movement image data.

The position optical image sensor140acquires a wide view position image of the writing surface surrounding the pen input device10, and derives from the position image position image data representing the position image. The position image data represents the intensity of the light measured at each photo detector of the position optical image sensor140, as discussed above. Each photo detector captures one picture element (pixel) of the image. The photo detectors can be, for example, photodiodes or phototransistors arranged in an array. The resolution of the position optical image sensor140is typically higher than the resolution of the movement optical image sensor80to enable the position detection system to determine the position of the writing tip20on the writing surface30with sufficient precision. In addition, the number of photo detectors in the position optical image sensor140is typically substantially greater than the number of photo detectors in the movement optical image sensor80.

The position optical image sensor140provides position image data, e.g., raw pixel values, to the processor100. The processor100processes the position image data and determines the position of the writing tip20of the pen input device10on the writing surface typically at the beginning and the end of each writing operation, as will be described in more detail below with reference toFIG. 3. In other embodiments, at least a portion of the processor100is incorporated in the same chip as the position image sensor140. The position data representing the position of the writing tip20on the writing surface can also be stored in the memory110for later processing and/or output to an external device via the I/O unit150.

The proximity sensor60, which can be embodied as a switch located in the pen body15, detects whether the writing tip20of the pen input device10is in contact with the writing surface30, as described above with reference toFIG. 1. The proximity sensor60provides a proximity signal to the processor100via a control line65. As described above, the state of the proximity signal indicates whether the writing tip20is in contact with the writing surface. The processor100responds to a change in the state of the proximity signal received on control line65to activate the movement detection system160and the position detection system170. When the state of the proximity signal changes from contact to non-contact, indicating that the user has moved the writing tip20away from contact with the writing surface, in one embodiment, the processor100deactivates the movement detection system160to save power, and momentarily activates the position detection system170to determine the position of the writing tip on the writing surface30at the end of the stroke.

The proximity sensor60, and other parts of the pen input device10(e.g., processor100, light source50and sensors80and140) are powered by an internal power supply90. In an embodiment, the power supply90comprises one or more rechargeable or non-rechargeable batteries. In other embodiments, power is supplied to the pen input device10via the I/O unit150from an external device, e.g., a computer. In embodiments in which the power supply90includes one or more rechargeable or non-rechargeable batteries contained within the pen body15, providing a power saving mode when the writing tip20is not in contact with the writing surface extends battery life.

FIG. 3is a block diagram illustrating the hardware and processing components of an exemplary embodiment of the pen input device10in accordance with the invention. As described above with reference toFIG. 2, the pen input device10includes a movement optical image sensor80and a position optical image sensor140, each for capturing images formed thereon and for generating respective image signals representing the images. The image signals are converted to corresponding image data by respective analog-to-digital converters (ADCs)85and145. For example, the ADCs85and145can be six-bit, eight-bit or ten-bit ADCs operating at a rate ranging from approximately 25 frames/sec to in excess of 1,000 frames/sec. Alternatively, one ADC (not shown) with suitable switching can be used instead of analog-to-digital conversion system85and145.

Movement image data202representing the series of movement images is output from the ADC85to the processor100, and position image data204representing each position image output from the ADC145is input to the processor100. The processor100includes a movement processor210for receiving the movement image data202and a position processor230for receiving the position image data204. In one embodiment, the movement image data202is received by a movement image processor212within the movement processor210. The movement image processor212processes the movement image data202and outputs processed movement image data in a form that enables the incremental changes in position of the writing tip20on the writing surface during each writing operation to be determined efficiently.

For example, in an embodiment in which the movement image sensor80is a color sensor incorporating a color filter array, the movement image processor212demosaics the image. Demosaicing is a process by which missing color values for each pixel location are interpolated from neighboring pixels. A number of demosaicing methods are known in the art; for example, pixel replication, bilinear interpolation and median interpolation. Other processing that the movement image processor212can perform includes noise filtering, image enhancement and extraction of features that are of interest.

In embodiments in which the movement image data is first processed by the movement image processor212, the processed movement image data is provided to a movement detection engine214within the movement processor210. In other embodiments, the unprocessed movement image data, e.g., raw pixel values, is input directly to the movement detection engine214. The movement detection engine214determines movement data216defining the changes between successive positions of the writing tip20on the writing surface30from the successive movement images captured during the writing operation. The movement data216is stored in the memory110for subsequent processing or is output to another application, e.g., a handwriting recognition application. A number of different processes are known that can be used by the movement detection engine214to determine the movement data216.

For example, as described in U.S. Pat. No. 6,172,354, entitled Operator Input Device, incremental movement is quantified by cross correlating a pattern in a previous movement image with the same pattern in a current movement image. Circular harmonics are used to determine movement of the current movement image relative to the previous movement image to provide rotationally-invariant incremental information. Another method for determining movement data216is described in U.S. Pat. No. 6,195,475, entitled Navigation System for Handheld Scanner. U.S. Pat. No. 6,195,475 models the correlation surface of correlation data representing a relationship between a sample frame and a reference frame as a general two-dimensional Taylor series expansion to quantify the magnitude and direction of displacement of a reference feature contained in both the reference frame and the sample frame.

A further movement detection process is taught in U.S. Pat. No. 5,578,813, entitled Freehand Image Scanning Device Which Compensates for Non-Linear Movement. The movement detection process described in U.S. Pat. No. 5,578,813 correlates successive frames of image data by comparing the successive frames. For example, as taught in U.S. Pat. No. 5,578,813, a reference frame and current frame of image data are acquired. The entire content of one of the frames (reference or current) is shifted in multiple trial shifts, in which each trial shift shifts each pixel in the frame in one of multiple directions. For example, the shifts may be in nine different directions, i.e., a “null” shift, a shift to the right, a shift to the right and down, etc.).

For each trial shift, those portions of the frames that overlap each other are subtracted on a pixel by pixel basis, and the resulting differences are typically squared and then summed to form a measure of similarity (correlation figure of merit) within that region of overlap. In addition, a correlation figure of merit for a “null” shift is also computed for the case where no movement has occurred between frames.

The shifting is accomplished by using address offsets for memories that can output an entire row or column of an array at one time. Dedicated arithmetic circuitry is connected to the memory array that contains the frame being shifted and to the memory array that contains the other frame to formulate the correlation figures of merit for each trial shift. Pixel locations along unmatched edges between the shifted frame and other frame are excluded when calculating the corresponding correlation figures of merit. The trial shift with the least difference (lowest correlation figure of merit) indicates the direction of movement between the two frames. The magnitude of movement in this example is one pixel unless the “null” shift has the lowest correlation figure of merit, in which case the magnitude of movement is zero.

The identity of the trial shift with the lowest correlation figure of merit provides the directional component of movement data216. The magnitude component of the movement data216is zero or one pixel, as described above. The current frame is then used as the reference frame for the next acquired image or the initial reference frame can be re-used to determine the incremental movement of successive current frames. In order to re-use the initial reference frame, the movement data (direction and displacement data) for the most recent frame is maintained for subsequent shifting and correlation figure of merit computation operations. Each shifting operation effectively throws away some of the reference frame, reducing the size of the reference frame and degrading the statistical quality of the correlation figures of merit. A new reference frame is taken when an edge of the shifted and reduced initial reference frame begins to approach the center of what was the original reference frame.

In other embodiments, fractional trial shifts equal to a fraction of the distance between adjacent pixels are used in addition to or in lieu of the whole pixel trial shifts described above. In addition, larger trial shifts are also possible. For example, in other modes of operation, the frame rates may be lower, requiring larger trial shifts to quantify the magnitude and direction of movement. However, the complexity of computing the correlation figures of merit increases as the trial shift amount increases.

The position image data204is received by a position image processor222or a position detection engine224within the position processor220. The position image processor222processes the position image data204and outputs processed position image data in a form that enables the position of the writing tip20on the writing surface30to be determined efficiently. For example, such image processing can include demosaicing, noise filtering, image enhancement and extraction of features (e.g., passive optical fiducials, such as previously-written strokes, on the writing surface) that are of interest.

In embodiments in which the position image data204is first processed by the position image processor222, the processed position image data is then provided to the position detection engine224within the position processor220. In other embodiments, the unprocessed position image data (e.g., raw pixel values) is input directly to the position detection engine224. The position detection engine224determines position data226representing the position of the writing tip20on the writing surface30typically at the beginning and the end of each writing operation. The position data226is stored in the memory110for subsequent processing or is output to another application (e.g., a character recognition application). For example, in one embodiment, the position detection engine224uses passive optical fiducials, such as the edges of the writing surface, as positional references with respect to which it calculates the position of the writing tip20on the writing surface. The position detection engine typically determines the positions of the writing surface at the beginning of each writing operation and at the end of the writing operation. The position detection engine may make additional determinations of the position of the writing tip one or more times during each writing operation, although this is not usually necessary and consumes power. The movement detection engine214and position detection engine224each include hardware, software and/or firmware capable of performing the functions described above. Software and/or firmware based embodiments can be programmed using any type of programming technique, including object oriented programming.

The processor100additionally includes a controller230connected the control line65to receive the proximity signal205from the proximity sensor60. The state of the proximity signal205indicates to the processor100whether the writing tip is in contact with the writing surface. For example, in one embodiment, when the proximity sensor60detects that the user has moved the writing tip20into contact with the writing surface, indicating the beginning of a writing operation, the proximity signal205goes high, and when the proximity sensor detects that the user has moved the writing tip20out of contact with the writing surface, indicating the end of the writing operation, the proximity signal205goes low. As noted above, the proximity sensor60detects whether either the writing tip20is touching the writing surface30or is within a defined proximity of the writing surface. In an embodiment in which the proximity sensor detects whether the writing tip20is within a defined proximity of the writing surface30, the proximity sensor may detect whether the writing tip20is close enough to the writing surface for the movement detection system160to determine movement of the writing tip relative to the writing surface.

The controller230controls the activation of the movement detection system160and the position detection system170based on the state of the proximity signal205. For example, in one embodiment, when the proximity signal205goes low, indicating that the state of the writing tip20has changed from contact to non-contact, the controller230deactivates the movement processor210and momentarily activates the position processor220to process a position image of the writing surface captured at the end of the writing operation. In addition, the controller230can additionally turn off the light source50(shown inFIG. 1) when the proximity signal205goes low, and can to turn on the light source when the proximity signal205goes high to further reduce power consumption.

In another embodiment, when the proximity signal205first goes high, indicating that the user has moved the writing tip20into contact with the writing surface30and a writing operation has begun, the controller230activates the position detection system170to obtain typically a single position image of the writing surface30to determine the position of the writing tip20on the writing surface. In addition, the controller230also activates the movement detection system160to obtain successive movement images from which the movement of the writing tip20is determined while the proximity signal205remains high, i.e., during the writing operation.

In one embodiment, the position data226is used to determine the position of the writing tip20on the writing surface at the beginning of each writing operation, i.e., when the movement detection system acquires the first of the series of movement images. The position image enables the position of the pen writing tip20on the writing surface30at the beginning of the writing operation, i.e., at the beginning of the stroke, to be determined. Thereafter, the movement data216is used to determine the magnitude and direction of movement (Δx, Δy) (e.g., the incremental changes in the position of the writing tip of the pen input device) between successive movement images acquired during the writing operation.

Movement detection is used to capture the movement of the writing tip20during each writing operation using a simple, inexpensive and low-power motion detector. However, as noted above, movement detection does not effectively capture the movement of the writing tip between strokes when the writing tip is out of contact with the writing surface. Thus, position detection is momentarily used to determine the position of the beginning of each stroke written on the writing surface to allow characters to be assembled with the strokes correctly positioned relative to one another. However, as a result of using movement detection in addition to position detection, the precision of the position detection system can be less than that of existing position detection systems, resulting in reductions in the cost and complexity of the position detection system. In addition, since the computational complexity needed to perform movement detection is orders of magnitude less than that required for position detection, using movement detection in lieu of full-time position detection while the writing tip is in contact with the writing surface results in considerable savings in both computation and power consumption.

The movement data provided by the movement detection system160is typically not absolute movement data, i.e., the movement represented by the movement data is not defined in a coordinate system that is fixed relative to the writing surface30. Instead, the movement data is relative movement data that depends on the orientation of the pen input device10relative to the edges of the writing surface30and the rotational orientation of the pen input device about its lengthways axis. Relative movement data does not define the orientation of the strokes relative to a fixed coordinate system. Variations in the orientation of the pen input device relative to the writing surface produce corresponding variations in the orientation of the strokes. To define the orientations of the strokes relative to a fixed coordinate system, two or more position detection operations are performed during each writing operation. For example, position detection operations are performed at the beginning and at the end of each writing operation, as described above.

In embodiments in which the movement data is absolute movement data or in embodiments which the relative movement data is converted into absolute movement data, no more than one position determination operation need be performed during each writing operation.

The controller230provides a link218that is used to associate the movement data216obtained during each writing operation performed by the pen input device (e.g., while the proximity signal205remains high) with the position data226for the writing operation. The link218is used by a subsequent application (e.g., a character recognition application) to link the position data226with the movement data216so that the subsequent application can assemble the strokes written by the user during successive writing operations to form characters that can be displayed on a screen or recognized by a character recognition application.

In assembling the characters, for example, the subsequent application locates the beginning of each stroke, as represented by the movement data, at the position of the beginning of the stroke indicated by the position data. Additionally, the subsequent application rotates the stroke, as represented by the movement data, about the beginning of the stroke to locate the end of the stroke at the position of the end of the stroke indicated by the position data.

In an embodiment, the movement data216and the position data226are provided to a character recognition engine (not shown) that operates on the movement data and the position data to assemble characters and to identify such characters. For each character, the handwriting recognition engine outputs a character code, such as an ASCII code, that can be used by a host (or processor) to store, display and/or otherwise process the character represented by the code. The character recognition engine may be located within the pen input device10or may be external to the pen input device.

FIG. 4shows a flow chart illustrating an exemplary process400for capturing the motion of the writing tip of a pen input device using a position detection system and a movement detection system. In block410, a determination is made of whether the writing tip is in contact with a writing surface. A “yes” result in block410indicates the beginning of a writing operation. When a “yes” result is produced in block410, a position image of the surface is acquired at block420and the position of the writing tip on the writing surface at the beginning of the writing operation is determined in block430. A “no” result in block410results in block410being repeated.

In block440, a movement image is acquired and is designated as a last movement image in block450. A new movement image is acquired in block460. In block470, the change in the position of the writing tip between the last movement image and the new movement image is determined as movement data. The movement data quantifies the magnitude and direction (Δx, Δy; or Δr, Δθ) of the change in the position of the writing tip between last movement image and the new movement image. In block480, another test is performed to determine whether the writing tip is still in contact with the writing surface. When the result in block480is “yes,” indicating that the writing tip is still in contact with the writing surface, execution returns to block450where the new movement image that was acquired in block460is designated as the last movement image, and another new movement image is acquired in block460. When the result in block480is “no,” indicating that the user has moved the writing tip out of contact with the writing surface, execution returns to block410via optional block490. In block490, an additional position image is captured and additional position data indicating the position of the writing tip on the writing surface at the end of the writing operation is derived from the additional position image. The method waits at block410until for user again to move the writing tip into contact with the writing surface.

In other embodiments, block420, in which the position image is acquired, and block430, in which the position data is determined, are performed after a number of movement images have been captured in blocks440-470. In this case, the position of the writing tip on the writing surface is determined after the beginning of the writing operation. In yet other embodiments, above-mentioned blocks420and430are each performed twice at arbitrary but spatially-separated points on the stroke. Each set of position data is linked to the movement data for the stroke.

FIG. 5schematically shows another exemplary embodiment of the pen input device in accordance with the invention. In the embodiment described above with reference toFIG. 3, the position detection system170uses two orthogonal edges of the writing surface30as a reference with respect to which it determines the position of the writing tip20on the writing surface. Under certain circumstances, the position detection system may fail to make a reliable position determination. For example, there may be insufficient contrast between the edges of the writing surface and the surface on which the writing surface is located for the edges of the writing surface to act as a positional references. This may occur, for example, when the writing surface30is white and is located on a light-colored surface. Although, this problem can be solved simply by placing the writing surface on a contrasting surface, such a contrasting surface may not always be available.

To solve this problem in a way that does not involve placing the writing surface30on a contrasting surface, in accordance with another embodiment of the present invention, and as shown inFIG. 5, the controller230delays activation of the position detection system270, composed of position image sensor140and the position processor220, until the user has completed one or more writing operations. For example, when the proximity signal205goes high, indicating that the user has moved the writing tip into contact with the writing surface, the controller230activates the movement detection system260to obtain movement data510for a first writing operation and to obtain movement data520for a second writing operation. The movement data510and520is temporarily stored in a buffer500. Once the controller230determines that the movement data510and520from the two writing operations are stored in the buffer500, the controller230activates the position detection system270to perform an operation in which a position image is acquired and converted to position image data and position data226is obtained using the marks made on the writing surface during the writing operations as positional references.

The position image data generated by the position detection system270includes position image data that represents the marks made by the writing tip20on the writing surface30in the two writing operations. The processor100uses the movement data510and520to identify the portion of the position image data that represents the marks, and additionally selects three, spatially-separated points on the marks to act as positional references. The processor100additionally constructs a positional coordinate system, such as a Cartesian coordinate system, using the three points as positional references. Finally, the processor uses the position data to define the positions of the marks represented by the movement data510and520in terms of the coordinate system to provide position data226for each of the marks. The processor transfers the movement data510and520from the buffer500to the memory110for storage therein as the movement data216. In addition, the controller230provides a link218to link the position data226with the movement data216.

As an example,FIGS. 6A and 6Billustrate exemplary writing operations performed by the pen input device10in accordance with the invention. The pen input device10performs a first writing operation610on the writing surface30. In the example shown, the first writing operation writes the letter “n.” The movement data510generated by the pen input device10during the first writing operation and temporarily stored in the buffer500(FIG. 5) represents changes between successive positions of the writing tip20on the writing surface30during the first writing operation.

Subsequently, the pen input device10performs a second writing operation620on the writing surface30. In the example shown, the second writing operation writes the letter “o.” The movement data520generated by the pen input device10during the second writing operation and temporarily stored in the buffer500(FIG. 5) represents the changes between successive positions of the writing tip20on the writing surface30during the second writing operation.

After the two writing operations are completed, position data is generated using marks written on the writing surface30by the two writing operations as positional references630. The positional references enable the position detection system270to determine the position of the pen input device10on the writing surface30.

Since the coordinate system established in response to the marks written by the first and second writing operations has axes that are not necessarily parallel to the direction in which writing is performed on the writing surface, the processor may later, after one or more lines of writing have been written, perform an operation similar to that described to realign the coordinate system so that one of the axes thereof is parallel to the direction of writing. Realigning the coordinate system simplifies and rationalizes the position determination process. After the coordinate system has been realigned, the existing position data is transformed to the new coordinate system.

FIG. 7is a flow chart illustrating another exemplary process700for determining the position of a pen input device using both position and movement data. Initially, at block710, a determination is made of whether the writing tip of the pen input device is in contact with the writing surface. A “no” result indicates that no writing operation is in process and execution loops back to block710.

A “yes” result in block710indicates the beginning of a writing operation and execution advances to block712. In block712, movement data representing the change between two successive positions of the writing tip on the writing surface is obtained. For example, the movement data may be obtained as described above with reference toFIG. 4. In block714, a test is performed to determine whether the writing tip is still in contact with the writing surface. A “yes” result in block714loops execution back to block712to obtain additional movement data. A “no” result in block714indicates that the writing operation has ended and execution advances to block716.

In block716, a test is performed to determine whether a first write operation flag has been set. A “no” result in block716indicates that the first write operation flag has not been set and that the write operation just completed is a first write operation. A “no” result in block716causes execution to advance to block718, where the first write operation flag is set, and then to block720, where the movement data is stored in the buffer500as first movement data510(FIG. 5). Execution then returns to block710to await the beginning of the second write operation.

A “yes” result is obtained in block716when the just-completed write operation is the second write operation. Execution advances to block722, where the first write operation flag is cleared, and then to block724, where the movement data is stored in the buffer500as second movement data520(FIG. 5).

Execution then advances to block726, where position image data is obtained, and then to block728, where position data is obtained using the marks made by the first and second writing operations as positional references. In block728, the position image data and the stored movement data are used to establish a coordinate system for position detection using the marks made by the first and second writing operations as positional references. Then, position data for the first and second movement data is obtained from the position image data. The position data defines the positions of the marks made by the first and second writing operations in the coordinate system.

The position data for each subsequent writing operation defines the position of the corresponding stroke in the coordinate system.

As described above, the pen input device10in accordance with the invention includes a proximity sensor that generates a proximity signal whose state indicates the state of contact of the writing tip with the writing surface. The proximity signal is used to indicate the state of contact to the controller230(FIG. 3).

In accordance with the invention, the proximity signal is additionally linked to a clock counter in a manner that causes the clock counter to generate a time stamp each time the proximity signal changes state. The time stamps generated at the beginning and end of each writing operation are linked to the movement data for the writing operation for use in character recognition. When such time stamps are available, the character recognition program that receives the data generated by pen input device10uses the time stamps to determine, for each writing operation, the time during which the writing tip is in contact with the writing surface and the time that elapses between the writing operation and the next writing operation. These times differ among characters but are typically consistent for a given writer. Consequently, the character recognition program can learn the characteristic times for the different characters written by a given writer.

In languages such as Chinese and Japanese, the order in which the strokes are written to form a given character is prescribed and the prescribed order is almost invariably followed by writers. The rhythm with which the writer writes the strokes to form the different characters, as defined by the time stamps, gives the character recognition program additional data that it can use to increase its recognition accuracy.

The innovative concepts described in the present application can be modified and varied over a wide range of applications. Accordingly, the scope of patented subject matter should not be limited to any of the specific exemplary teachings discussed, but is instead defined by the following claims.