Patent Description:
Computing devices can include a touchscreen display that presents visual output and receives touch input. Some computing devices can include a camera inside the touchscreen display. Accurately processing touch input near the camera can be difficult. <CIT> discloses a method of compensated touch data values including acquiring touch data values about a dead sensing zone of a touch screen and determining a peak value of those touch data values. <CIT> discloses an interactive control method performed at a terminal.

According to an example, a non-transitory computer-readable storage medium can comprise instructions stored thereon. When executed by at least one processor, the instructions can be configured to cause a computing device including a camera included behind a touchscreen, to receive a.

According to an example not being part of the invention, a non-transitory computer-readable storage medium can comprise instructions stored thereon. When executed by at least one processor, the instructions can configured to cause a computing device to determine that a measured location of a touch input on a touchscreen is within a predefined area proximal to a camera (that is, meeting a proximity criterion with respect to the camera), determine a shift value for the touch input based on the measured location of the touch input, determine a shifted location for the touch input based on the measured location and the shift value, and process the touch input based on the shifted location. In practice, an area where an object such as a stylus or a user's finger contacts a touchscreen, may include a plurality of locations with touch inputs (i.e. non-zero measurements of applied force). Some of these may be within the predefined area and some may not be. The shift value may be applied to those of the measured locations which are within the area. Subsequently, an operation may be applied (e.g. based on the shifted locations and the measured values outside the predefined area) to estimate a location of the center of the area where the object touched the touchscreen.

According to an example not being part of the invention, a method can comprise contacting a touchscreen at multiple known locations, the multiple known locations including at least a first location with a first density of touch sensors and a second location with a second density of touch sensors, the second density of touch sensors being less than the first density of touch sensors, determining multiple measured locations on the touchscreen corresponding to the multiple known locations, and generating a map, the map mapping the multiple measured locations to the multiple known locations.

According to an example not being part of the invention, a non-transitory computer-readable storage medium can comprise instructions stored thereon. When executed by at least one processor, the instructions can be configured to cause a computing device to receive multiple touch inputs, the multiple touch inputs being received at different times and being included in a single moving contact, determine that a measured location of at least one of the multiple touch inputs is skewed, based on determining that the location of at least one of the multiple touch inputs is skewed, correct the skewed location of the at least one of the multiple touch inputs based on the location of at least one of the multiple touch inputs that is skewed and locations of at least two other of the multiple touch inputs, and process the multiple touch inputs with the corrected location.

In further examples, computer systems are provided comprising a processor and a computer-readable storage medium of any of the types defined above, and arranged to perform the instructions.

A touchscreen can have lower density of touch sensors near and/or over a camera to improve the camera's ability to capture images. The lower density of touch sensors near the camera can cause measured touch values to be lower than for locations farther from the camera with higher densities of touch sensors. A computing device can generate a compensated touch value based on the measured touch value and a scaling value. The scaling value can be associated with the location of touch input for which the touch value was measured. The density of touch sensors is a number of touch sensors per unit area of the touchscreen; for example, the density of touch sensors at any location may be defined as the number of touch sensors in a circle, rectangle or square of predetermined size and/or orientation centered at that location.

The lower density of sensors, and/or the transition between locations with different densities of touch sensors, can cause measured locations of touch inputs to be shifted from the actual and/or true locations of the touch inputs. The computing device <NUM> can determine shifted locations based on the measured locations and either shift values or shifted locations associated with the measured locations. In some examples, a different computing system can determine the shift values or shifted locations by contacting the touchscreen with a stylus at multiple known locations and mapping the known locations at which the stylus contacted the touchscreen to the measured touch locations.

The lower density of sensors, and/or the transition between locations with different densities of touch sensors, can cause measured locations of touch inputs to be skewed from the actual and/or true locations of the touch inputs. The computing device can determine that the measured locations are skewed by comparing the measured locations and/or shifted locations to other measured locations and/or shifted locations within a continuous contact. The computing device can correct the skewed locations based on the other measured locations, such as by applying a filter to the multiple locations.

<FIG> shows a computing device <NUM> according to an example implementation. The computing device <NUM> can include, for example, a smartphone with a touchscreen <NUM>, a tablet computing device with the touchscreen <NUM>, a notebook or laptop computing device with the touchscreen <NUM>, or a personal computing device with the touchscreen <NUM>.

The touchscreen <NUM> can present visual output to a user. The touchscreen <NUM> can include pixels that emit colored light to collectively display pictures, icons, letters, or other images.

The touchscreen <NUM> can also receive touch input from the user. Touch input can include physical contact on the touchscreen <NUM> that causes a measurable change, such as a change in capacitance and/or change in resistance, within the touchscreen <NUM>. In some examples, the touchscreen <NUM> can include a capacitive touchscreen that detects touch input by measuring changes in capacitance caused by part of a human body, such as the user's finger, contacting the touchscreen <NUM>. In some examples, touch sensors included in the touchscreen <NUM> can include metal traces surrounding the pixels. In some examples, the touchscreen <NUM> can detect touch input by measuring changes in resistance.

The computing device <NUM> can include a frame <NUM> supporting, enclosing, and/or surrounding the touchscreen. The frame <NUM> can also support, enclose, and/or surround other components of the computing device <NUM> not shown in <FIG>, such as a memory, a processor, a speaker, a microphone, a wired communication interface, and/or a wireless communication interface, as non-limiting examples.

The computing device <NUM> can include a camera <NUM>. The camera <NUM> can capture visual data from outside the computing device <NUM>. The camera <NUM> can be included in and/or behind the touchscreen <NUM>, and arranged to capture the visual data from light passing through the touchscreen <NUM>.

<FIG> shows a front portion of the computing device <NUM>, which includes the touchscreen <NUM> and the camera <NUM>. In some examples, the computing device <NUM> can include a back portion, opposite from the front portion, which also includes a second camera but does not include a touchscreen or other display.

To allow the touchscreen <NUM> to present visual output in the area over the camera <NUM>, but still allow the camera <NUM> to capture visual data, pixel density in an area of the touchscreen <NUM> over the camera <NUM> can be lower than pixel density in areas of the touchscreen <NUM> that are not over the camera <NUM>. Here the term "over the camera" refers to a portion of the touchscreen for which the normal direction intercepts the camera. For some such portions of the touchscreen, the normal direction may intercept a light-capturing region of the camera. The lower pixel density allows the camera <NUM> to capture visual data sufficiently accurately, and presents visual output of satisfactory quality. However, the lower pixel density can cause non-uniformity of parasitic capacitance, which is measured to determine touch inputs, compared to other parts of the touchscreen. <FIG>, described below, shows an expanded area <NUM> of the touchscreen <NUM> over and around the camera <NUM>, with some portions of the touchscreen <NUM> in the expanded area <NUM> having lower pixel density than other portions of the touchscreen <NUM>.

<FIG> shows a partial cross-sectional view of the computing device <NUM> shown in <FIG> according to an example implementation. The cross-sectional view is along the dashed line denoted 'A' in <FIG>.

The touchscreen <NUM> can include a cover <NUM>, a touch sensor layer <NUM>, and/or a display layer <NUM>. The cover <NUM> can be made of a transparent material such as glass or plastic. The cover <NUM> can cover and/or protect other components of the touchscreen <NUM>, such as the touch sensor layer <NUM> and/or the display layer <NUM>. The cover <NUM> can superpose, and/or be disposed above, other components of the touchscreen <NUM>, such as the touch sensor layer <NUM> and/or the display layer <NUM>.

The touchscreen <NUM> can include the touch sensor layer <NUM>. The touch sensor layer <NUM> can detect and/or measure touch input. The touch sensor layer <NUM> can detect and/or measure touch input, such as by detecting and/or measuring changes in capacitance and/or changes in resistance in response to the touch input. In some examples, the touch sensor layer <NUM> can be disposed and/or located between the cover <NUM> and the display layer <NUM>. In some examples, portions of the touch sensor layer <NUM> can surround portions of the display layer <NUM>, such as by metal traces included in the touch sensor layer <NUM> surrounding display pixels included in the display layer <NUM>.

The touchscreen <NUM> can include the display layer <NUM>. The display layer <NUM> can generate and/or display visual output, such as graphical output. The display layer <NUM> can include multiple pixels that each generate colored light to collectively display pictures, icons, letters, or other images.

The computing device <NUM> can include the camera <NUM>. In some examples, the camera <NUM> can be disposed and/or located below the touchscreen <NUM> (that is, in the normal direction to the cover <NUM> towards the display layer <NUM>). In some examples, the camera <NUM> can be disposed and/or located between the touchscreen <NUM> and the frame <NUM>. In some examples, the camera <NUM> can be disposed and/or located within the touch sensor layer <NUM> and/or display layer <NUM> of the touchscreen <NUM>.

The computing device <NUM> can include the frame <NUM>. The frame <NUM> can support, surround, and/or enclose components of the computing device <NUM>, such as the touchscreen <NUM> and/or the camera <NUM>.

<FIG> shows the expanded area <NUM> of the touchscreen <NUM> and the camera <NUM> included in the computing device <NUM> shown in <FIG> according to an example implementation. The expanded area <NUM> includes multiple locations <NUM>-<NUM>. As denoted in <FIG>, the second digit in the reference number for the locations <NUM>-<NUM> indicates the row of the respective location, and the third digit in the reference number denotes the column. While <FIG> divides the expanded area <NUM> into sixty-four locations, other levels of granularity can be implemented, and the expanded area can be divided into more or fewer than sixty-four locations.

In this example, the shaded locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> have lower densities of touch sensors than the other locations in the expanded area <NUM>. A density of touch sensors can represent a number of touch sensors per unit of area of the touchscreen <NUM> that superposes the touch sensors, and/or a number of touch sensors under a given area of the touchscreen <NUM>. In some examples, the lower densities of touch sensors can include a lower density of pixels and/or a lower density of metal traces, surrounding the pixels, of which changes in capacitance are measured. In some examples, the shaded locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> with lower densities of touch sensors can be more proximal to, and/or closer to, the camera <NUM> than the other locations in the expanded area <NUM>. The term "proximal" is used to mean "meeting a proximity criterion". For example, the proximity criterion may be that the location is within a predetermined distance of the camera, or that it overlies the camera (i.e. a normal to the sensor layer at the location intercepts the camera), or that the location is within a predetermined distance in the plane of the sensor layer <NUM> from a portion of the sensor layer <NUM> which lies on an optical axis of the camera. The change in density can be abrupt, with adjacent locations having much higher or lower densities of touch sensors than each other, or gradual, with a path of locations gradually increasing or decreasing in density of touch sensors. In some examples, the other locations with higher densities of touch sensors can be farther from, and/or less proximal to, the camera <NUM> than the locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. In some examples, the locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> with lower density of touch sensors can fully superpose the camera <NUM> (i.e. entirely cover the camera <NUM>, as viewed in the direction transverse to the cover <NUM>).

<FIG> shows a finger <NUM> contacting a portion of the expanded area <NUM> near the camera <NUM> according to an example implementation. The finger <NUM> can contact any number of locations <NUM>-<NUM> inside or outside the expanded area <NUM>.

<FIG> shows measured touch values for locations in the expanded area <NUM> that were contacted by the finger <NUM> according to an example implementation. <FIG> shows a map <NUM> of measured touch values corresponding to the locations <NUM>-<NUM> shown in <FIG> and <FIG>. For ease of viewing, the reference numbers of the locations <NUM>-<NUM> to which the measured touch values correspond are not shown in <FIG>. However, the locations with lower densities of touch sensors are shaded for ease of reference. While <FIG> shows the measured touch values graphically in a two-dimensional map, the measured touch values can be represented and/or stored in other formats, such as in a comma-separated value file or a two-dimensional array. Measured touch values can represent raw data received from touch sensors, such as changes in capacitance and/or changes in resistance. The measured touch values can include a range of numerical values, such as a lower bound of zero (<NUM>) to indicate no contact or touch was detected and an upper bound to represent the maximum measurable force or change of capacitance and/or resistance. A measured location of a touch input can represent a location at which a measured touch value is greater than zero (<NUM>).

In the example shown n <FIG>, the areas with values of <NUM> indicate that no contact or touch was measured or detected at the locations corresponding to areas with values of <NUM>. Also in this example, the areas corresponding to locations <NUM>, <NUM>, <NUM>, and <NUM> have measured touch values of <NUM>. Also in this example, the areas corresponding to locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM> have measured touch values of <NUM>. These measured touch input values are examples, and the computing device <NUM> could measure other touch input values. Based only on these raw touch input data, the contact by the finger <NUM> would appear to be placed most heavily toward the left, along the second column, as reflected by the measured touch values of fifty for locations <NUM>, <NUM>, <NUM>. However, as discussed above, the locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> have lower densities of touch sensors than other locations, which can cause the measured values at the locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> to be lower than the actual contact by the finger <NUM> should reflect. To correct for inaccurately low measured touch values, the computing device <NUM> can compensate for the lower density of touch sensors by increasing the touch values for the locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>.

<FIG> shows a scaling map <NUM> (or "scaling value map") for locations in the expanded area according to an example implementation. A scaling map can associate scaling values with locations. The locations can be identified by x-values and y-values where the locations represent points on the touchscreen <NUM>, and/or by ranges of x-values and y-values where the location represents a rectangular area on the touchscreen <NUM>. The scaling map <NUM> shows scaling values by which the measured touch values will be multiplied to compensate for the lower density of touch sensors in the locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> that are proximal to, and/or near, the camera <NUM>. Scaling values can be values associated with locations that will correct for lower densities of touch sensors, so that a contact by the same object with the same amount of force will, after correction based on the measured touch value and the scaling value, have the same compensated touch value in different locations, even if the different locations have different densities of touch sensors.

In this example, the scaling values for the locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> that are proximal to, and/or near, the camera <NUM>, are two (<NUM>), and the scaling values for other locations is one (<NUM>). This is merely an example, and other scaling values can be included and/or stored in the scaling value map <NUM> to compensate for the lower density of touch sensors in some locations. The scaling values can include two values, in the example shown in <FIG>, or the scaling values can be more granular, with many more than two distinct values represented on the scaling map. While <FIG> shows the scaling value map <NUM> graphically in a two-dimensional map, the scaling value map <NUM> can be represented and/or stored in other formats, such as in a comma-separated value file or a two-dimensional array. In some examples, the scaling values for each location can be inversely proportional to the density of touch sensors in the respective location, and/or a ratio of scaling values of two different locations to each other can be inversely proportional to a ratio of the densities of the touch sensors in those same two different locations to each other.

The computing device <NUM> can correct and/or compensate for the lower density of touch sensors in locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> by multiplying the measured touch values shown in <FIG> by the scaling values in <FIG>. The product of multiplying the measured touch values shown in <FIG> by the scaling values in <FIG> can be compensated touch values.

<FIG> shows compensated touch values for locations in the expanded area <NUM> that were contacted by the finger according to an example implementation. The compensated touch values are represented in <FIG> within a compensated touch value map <NUM> (or "compensated value map"). While <FIG> shows the compensated value map <NUM> graphically in a two-dimensional map, the compensated value map <NUM> can be represented and/or stored in other formats, such as in a comma-separated value file or a two-dimensional array. Compensated touch values can be numerical values with a same range as the measured touch values that represent force applied by an object after correcting and/or compensating the measured touch value by the scaling value. Compensated touch values can be passed to, and/or processed by, an application and/or operating system as touch input to be handled by the application and/or operating system.

The computing device <NUM> can process the compensated touch values as the touch input to determine how to respond to the touch input. The compensated touch value of <NUM> for location <NUM>, and compensated touch values of <NUM> for locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> indicate that the contact by the finger <NUM> was strongest and/or heaviest at location <NUM>, and was also significant for locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>.

<FIG> is a graph showing measured touch values <NUM> and compensated touch values <NUM> as a function of location according to an example implementation. The graph of <FIG> shows touch values as a function of location in only a single direction, such as either vertical or horizontal. The smooth curves of the measured touch values <NUM> and compensated touch values <NUM> shown in <FIG> show greater granularity of measured touch values, and/or division of the touchscreen <NUM> and/or expanded area <NUM> into more locations, than shown in the preceding figures. Furthermore, in this example there may be different scaling values for different corresponding locations within the lower density area e.g. with the scaling value rising towards the centroid of the lower density area <NUM>. The difference between the compensated touch values <NUM> and the measured touch values <NUM> in the lower density area <NUM>, which can correspond to the locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> that have lower density of touch sensors, indicates the effect of multiplying the measured touch values by the scaling values. The skewed curve of the measured touch values <NUM> would incorrectly show a touch contact that was strong on the right side of the contact. The symmetrical curve of the compensated touch values <NUM> correctly shows a touch contact that is strongest in a center of the contact.

In the example shown in <FIG>, the skewed curve of the measured touch values <NUM> shows a measured center <NUM>, at a peak of the measured touch values curve <NUM>, that is skewed and/or shifted to the right of the actual center <NUM> of the contact, as shown by the peak of the compensated touch values <NUM> curve. The computing device <NUM> can determine an accurate location of a center of a contact, at the actual center <NUM> and/or peak of the compensated touch values <NUM> curve, by compensating the measured touch values based on the compensated touch value map <NUM>.

<FIG> shows a predefined area <NUM> within the expanded area <NUM> in which measured locations will be shifted according to an example implementation. The predefined area <NUM> can be an area on the touchscreen <NUM> in which measured locations can be shifted and/or incorrect due to change in density of touch sensors. The predefined area <NUM> can be stored in the computing device <NUM>, such as in a memory of the computing device <NUM>. In some examples, the predefined area <NUM> is on and/or at an edge of the locations with lower density of touch sensors. In some examples, the predefined area <NUM> is at an interface between the locations with lower density of touch sensors and the locations with the normal and/or standard density of touch sensors.

In some examples, the predefined area <NUM> has a shape that is homeomorphic with an annulus. Two shapes can be considered to be homeomorphic if the shapes can be morphed into each other by bending or stretching their respective portions without breaking, cutting or attaching any of their respective portions. The predefined area <NUM> can define a hole <NUM> which is not part of the predetermined area <NUM>, and/or the hole <NUM> can be bounded by the predefined area <NUM>. The predefined area <NUM> can have a shape surrounding the hole <NUM> such as a square, a circle, a rectangle, or an ellipse, as non-limiting examples. The hole <NUM> can superpose at least a portion of the camera <NUM> (not shown in <FIG>).

In some examples, the predefined area <NUM> has a shape that is homeomorphic with a square, such as a square, a circle, a rectangle, or an ellipse, as non-limiting examples. The predefined area <NUM> can superpose at least a portion of, and/or all of, the camera <NUM>.

In some examples, the transition from lower density of touch sensors to higher density of touch sensors within the expanded area of the touchscreen <NUM> within the predefined area <NUM> can cause measured touch locations to be shifted outward, and/or away from the camera <NUM>, compared to the actual locations of the contacts from which the touch locations are measured. The computing device <NUM> can correct the outward shift by shifting the measured locations inward toward the camera <NUM>. The computing device <NUM> can shift the measured locations inward by adding a vector, such as an x-value and a y-value, to the measured touch location. The computing device <NUM> can determine the vector to add to the measured touch location by mapping the measured touch location to a vector. The measured touch locations and vectors can be associated with each other on a stored map and/or table, a comma separated value file, or a two-dimensional array, as non-limiting examples.

<FIG> shows shift values for correcting measured locations according to an example implementation. Shift values can be vectors by which measured touch values will be changed to arrive at corrected and/or shifted values. In this example, the shift values are shown graphically on a shift map <NUM>. The bullet holes show measured locations for which no shift is needed. The arrows show a shift value graphically. The shift values can also be represented as a pair of values, such as a positive or negative value for each of a horizontal (x) direction and a vertical (y) direction.

The computing device <NUM> can add the shift values to the measured touch locations to determine correct, shifted locations. The computing device <NUM> can determine the shift values based on the measured touch locations, such as by mapping the measured touch values to the shift locations. The computing device <NUM> can process the touch input based on the shifted values and/or shifted locations. Shifted values and/or shifted locations can be locations for which touch inputs will be processed after correcting for shift caused by the change in density of touch sensors. In some examples, the computing device <NUM> can determine shifted locations and/or corrected locations directly based on the measured touch locations, such as by mapping the measured touch locations to corrected locations and/or shifted locations. In practice, an area where an object such as a stylus or a user's finger, may include a plurality of locations with non-zero touch values. Some of these may be within the predefined area and some may not. The shift value may be applied to those of the measured locations which are within the area. Subsequently, an operation may be applied (e.g. based on the shifted locations and the (unshifted) measured values outside the predefined area) to estimate a location of the center of the area where the object touched the touchscreen. For example, the estimated location may be a weighted average of the shifted and unshifted locations, where the weights are the respective touch values or the compensated touch values. Or the touch values (or the compensated touch values) at the shifted and unshifted locations may be used in a curve fitting process, to find a peak of the curve.

The computing device <NUM>, and/or another computing system, can determine the shift values experimentally. In some examples, a computing system can cause a stylus or other object to contact the touchscreen <NUM>, or another touchscreen with similar hardware features as the touchscreen <NUM> and/or computing device <NUM>, at multiple known locations on the touchscreen. The computing system can map the known touch locations to corresponding measured touch locations, and/or associate the known touch locations to the corresponding measured touch locations. The computing system can determine differences between the known touch locations and the corresponding measured touch locations. The computing system can store the differences between the known touch locations and the corresponding measured touch locations as shift values in association with the corresponding measured touch locations. The computing system can store the shift values in association with the corresponding measured touch locations as a shift map <NUM> as shown in <FIG>, or in other formats, such as a table, a comma separated value file, or a two-dimensional array, as non-limiting examples.

<FIG> shows an arm <NUM> causing a stylus <NUM> to contact the expanded area <NUM> according to an example implementation. The arm <NUM> and/or stylus can be controlled by a computing system that causes the stylus <NUM> to contact the touchscreen <NUM>, and/or the expanded area <NUM> within the touchscreen <NUM>, in multiple known locations. The known locations can be locations on the touchscreen <NUM> which are previously determined and/or where the stylus <NUM> is intended to contact the touchscreen <NUM>, and/or where instruments, external to the computing device <NUM> and/or computing system that the stylus <NUM> is contacting, measure the stylus as contacting. The locations on the touchscreen <NUM> at which the stylus <NUM> contacts the touchscreen <NUM> can be known to the computing system, and/or predetermined. The computing system can determine shift values based on differences between the known locations of the contacts and the measured touch locations of the contacts. The computing system can store the determined shift values, such as in a shift map <NUM> as shown in <FIG>, or in other formats, such as a table, a comma separated value file, or a two-dimensional array, as non-limiting examples. In some examples, the computing system can generate the shift map <NUM> based on the known locations of the contacts and the measured touch locations of the contacts. The computing system, and/or an intermediary computing system, can provide the stored shift values to the computing device <NUM>.

In some examples, the measured touch locations can be measured centers of the contacts, such as the measured center <NUM> shown in <FIG> that results from skewing caused by reduced sensor density in some areas. In these examples, the computing device <NUM> can shift locations of a measured center <NUM> to an actual center based on the shift map <NUM>.

<FIG> shows measured touch locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and corrected touch locations <NUM>, <NUM> according to an example implementation. The computing device <NUM> can receive multiple touch inputs at different times. The computing device <NUM> can receive the multiple touch inputs within a predetermined time period, such as within one second of each other. This can done using respective recorded times associated with the measured touch locations. The multiple touch inputs can be part of, and/or included in, a single moving contact across the touchscreen <NUM> and/or expanded area <NUM> of the touchscreen <NUM>. The measured touch locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> can represent measured locations of the multiple touch inputs. In some examples, the change to the density of the touch sensors can cause some of the measured locations to be skewed from the actual locations of the contacts. In some examples, the change to the density of the touch sensors can cause some of the measured locations to be skewed from the actual locations of the contacts even after performing the shifts described above with respect to <FIG>, <FIG>, and <FIG>.

The computing device <NUM> can determine that a location of at least one of the multiple measured touch locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> is skewed. In the example shown in <FIG>, the computing device <NUM> determines that measured touch locations <NUM>, <NUM> are skewed. The distance between the locations <NUM>, <NUM> and corrected locations <NUM>, <NUM> is exaggerated in <FIG> for illustrative purposes. The computing device <NUM> can determine that at least one of the multiple measured touch locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> is skewed, based on comparing the locations and/or times of the measured touch locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> to each other (here the phrase "the time of a measured touch location" means a time at which force was measured at the location; this time may be recorded), such as by comparing the locations and/or times of the measured touch locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> to predicted curves and/or arcs, and/or by applying one or more filters to the locations and/or times of the measured touch locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and determining that the location of at least one of the measured touch locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> is skewed.

Based on determining that the location of at least one of the measured touch locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> is skewed, the computing device <NUM> can correct the measured location of the touch input that is determined to be skewed, such as the measured touch locations <NUM>, <NUM>, based on the location of the touch value and/or measured touch location <NUM>, <NUM> and at least two other touch values, such as the measured touch locations <NUM>, <NUM>, <NUM>, <NUM>. In some examples, the computing device <NUM> can correct the skewed location(s) by applying a filter to the measured touch locations <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. In some examples, the filter can include an infinite impulse response filter, a Kalman filter, and/or a Butterworth filter.

In the example shown in <FIG>, vectors <NUM>, <NUM> represent corrections to the measured touch locations <NUM>, <NUM>. The computing device <NUM> can add the vectors <NUM>, <NUM> to the measured touch locations <NUM>, <NUM>, to arrive at the corrected locations <NUM>, <NUM> and/or a corrected path. The computing device <NUM> can process the touch inputs based on the corrected locations <NUM>, <NUM> and/or corrected path <NUM> and the measured touch locations <NUM>, <NUM>, <NUM>, <NUM> that are determined not to have been skewed.

<FIG> is a schematic diagram of the computing device <NUM> according to an example implementation. The computing device <NUM> can include a touch input processor <NUM>. The touch input processor <NUM> can process touch input, such as by measuring changes in capacitance of touch sensors caused by objects contacting the touchscreen <NUM>.

The touch input processor <NUM> can include a touch value measurer <NUM>. The touch value measurer <NUM> can measure values associated with touch input to the touchscreen <NUM>, such as location, magnitude, and/or time.

A location measurer <NUM> included in the touch value measurer <NUM> can measure a location of the touch input. The location can be measured based on which touch sensors experienced changes in capacitance. The location can be expressed and/or stored as an x-value and a y-value, which can represent a horizontal distance from a corner of the touchscreen <NUM> and a vertical distance from the same corner of the touchscreen <NUM>. In some examples, the horizontal distance can be measured in pixels and/or the vertical distance can be measured in pixels.

A magnitude measurer <NUM> included in the touch value measurer <NUM> can measure a magnitude and/or force of the touch input. The magnitude can be measured based on an amount of the change in capacitance experienced by the touch sensors. The magnitude can be expressed and/or stored as an absolute value.

A time measurer <NUM> included in the touch value measurer <NUM> can measure a time and/or duration of the touch input. The time can be measured by comparing the change(s) in capacitance to a clock inside, and/or accessed by, the computing device <NUM>. The time can be expressed and/or stored as an absolute date and/or time, or relative to a specific time. The duration can be a time between when the touch input began and when the touch input ended. The duration can be expressed and/or stored as seconds and/or fractions of seconds.

The computing device can include a touch value compensator <NUM>. The touch value compensator <NUM> can compensate, enhance, and/or multiply measured touch values by a scaling value, which is based on, and chosen to compensate for, fewer touch sensors being at the location of a touch contact.

The touch value compensator <NUM> can include a scale mapper <NUM>. The scale mapper <NUM> can map a location of a touch contact to a scaling value, such as a scaling value included in the scaling value map <NUM> shown in <FIG>. The touch value compensator <NUM> can multiply the measured touch value by the scaling value to generate a compensated touch value.

The computing device <NUM> can include a shift corrector <NUM>. The shift corrector <NUM> can correct shifted touch location values that are caused by changes in the touch sensor density, such as shifted touch location values in the predefined area <NUM> described above with respect to <FIG>.

The shift corrector <NUM> can include a shift mapper <NUM>. The shift mapper <NUM> can map detected and/or measured touch locations to shift values. The shift mapper <NUM> can, for example, determine whether the measured location of the touch input is within the predefined area <NUM>. If the measured location is within the predefined area, the shift mapper <NUM> can determine a shift value based on the measured touch location. In some examples, the shift mapper <NUM> can determine the shift value by, for example, mapping the measured touch location to the shift value included on the shift map <NUM>. In some examples, the shift mapper <NUM> can determine the shift value by finding a shift value, stored in a file, that is associated with the measured touch location. In some examples, instead of a shift value, the shift corrector <NUM> can determine a corrected location for the touch input by finding a corrected location, stored in a file, that is associated with the measured touch location.

The shift corrector <NUM> can determine a corrected and/or shifted location for the touch input based on the measured location and the shift value. The computing device <NUM> can process the touch input based on the shifted location and/or corrected location.

The computing device <NUM> can include a skew corrector <NUM>. The skew corrector <NUM> can correct skewed locations where multiple touch inputs indicated that a location of at least one of the touch inputs is skewed. In some examples, the skew corrector <NUM> can correct the skewed locations after the touch value compensator <NUM> has corrected and/or compensated the measured touch value and/or the shift corrector <NUM> has corrected the shifted location.

The skew corrector <NUM> can include a skew determiner <NUM>. The skew determiner <NUM> can determine that a location of at least one of multiple touch inputs is skewed. The skew determiner <NUM> can determine that the location of at least one of the multiple touch inputs is skewed based on the times and locations of the multiple touch inputs. The skew determiner <NUM> can, for example, determine that the location of at least one of the multiple touch inputs is skewed based on the multiple touch inputs not fitting a previously-stored pattern, such as an arc or a line.

The skew corrector <NUM> can include a filter <NUM>. The filter <NUM> can correct the location of the touch input location that the skew determiner <NUM> determined was skewed. The filter <NUM> can correct the location of the skewed touch input location by applying a filter to the multiple touch inputs, such as an infinite impulse response filter, a Kalman filter, and/or a Butterworth filter.

The computing device <NUM> can include a display processor <NUM>. The display processor <NUM> can control the graphical output generated by the touchscreen <NUM>, such as based on instructions from one or more applications executing on the computing device <NUM> and/or an operating system executing on the computing device <NUM>.

The computing device <NUM> can include a camera processor <NUM>. The camera processor <NUM> can receive and/or process visual data received by the camera <NUM>.

The computing device <NUM> can include at least one processor <NUM>. The at least one processor <NUM> can execute instructions, such as instructions stored in at least one memory device <NUM>, to cause the computing device <NUM> to perform any combination of methods, functions, and/or techniques described herein.

The computing device <NUM> may include at least one memory device <NUM>. The at least one memory device <NUM> can include a non-transitory computer-readable storage medium. The at least one memory device <NUM> can store data and instructions thereon that, when executed by at least one processor, such as the processor <NUM>, are configured to cause the computing device <NUM> to perform any combination of methods, functions, and/or techniques described herein. Accordingly, in any of the implementations described herein (even if not explicitly noted in connection with a particular implementation), software (e.g., processing modules, stored instructions) and/or hardware (e.g., processor, memory devices, etc.) associated with, or included in, the computing device <NUM> can be configured to perform, alone, or in combination with the computing device <NUM>, any combination of methods, functions, and/or techniques described herein.

The memory <NUM> can store the maps, such as the scaling map <NUM> and/or shift map <NUM> described above. The memory <NUM> can store touch values <NUM>, such as touch values described above. For each stored touch value <NUM>, the memory <NUM> can store a location <NUM>, a magnitude <NUM>, and/or a time <NUM> and/or duration.

The computing device <NUM> may include at least one input/output node <NUM> (that is, interface). The at least one input/output node <NUM> may receive and/or send data, such as from and/or to, a server, and/or may receive input and provide output from and to a user. The input and output functions may be combined into a single node, or may be divided into separate input and output nodes. The input/output node <NUM> can include, for example, the touchscreen <NUM>, the camera <NUM>, a speaker, a microphone, one or more buttons, and/or one or more wired or wireless interfaces for communicating with other computing devices.

<FIG> shows a pipeline <NUM> of functions performed by the computing device <NUM> according to an example implementation. The computing device <NUM> can receive touch input (<NUM>). The computing device <NUM> can receive the touch input via the touchscreen <NUM>.

The computing device <NUM> can process a measured touch value (<NUM>). The computing device <NUM> can process the measured touch value by determining a magnitude of the touch value based, for example, on an amount of change to the capacitance of a touch sensor that detected the touch input.

The computing device <NUM> can compensate the measured touch value (<NUM>). The computing device <NUM> can compensate the measured touch value by, for example, determining a scaling value associated with a location of the touch input, and multiplying the measured touch value by the scaling value.

The computing device <NUM> can determine whether the location of the touch input is within the predefined area <NUM> (<NUM>). If the touch input is located within the predefined area <NUM>, then the computing device <NUM> can shift the measured location of the touch input (<NUM>), such as by shifting the measured location of the touch input based on the shift map <NUM>.

The computing device <NUM> can reduce jitter of the measured touch input (<NUM>). The computing device <NUM> can reduce the jitter of the measured touch input by, for example, applying a filter to multiple touch inputs, such as an averaging filter.

The computing device <NUM> can determine whether one or more locations, which may or may not have been shifted at (<NUM>), are skewed (<NUM>). The computing device <NUM> can determine whether the locations are skewed based on the locations and times of multiple touch input, such as by applying a filter to the multiple locations and times. If the computing device determines that the locations are not skewed, then the computing device <NUM> can process the touch values (<NUM>). If the computing device <NUM> determines that the locations are skewed, then the computing device <NUM> can correct the skew (<NUM>) and then process the corrected touch values (<NUM>).

<FIG> is a flowchart showing a method performed by the computing device <NUM> according to an example implementation. The method can include receiving a first measured touch value (<NUM>). The first measured touch value can indicate a first touch input at a first location, such as location <NUM>, near a camera <NUM> included in the computing device <NUM>. Examples of measured touch values are shown in <FIG>. The method can also include receive a second measured touch value (<NUM>). The second measured touch value can indicate second touch input at a second location, such as location <NUM>, farther from the camera <NUM> than the first location. A density of touch sensors in the second location can be greater than a density of touch sensors in the first location. The method can also include generating a first compensated touch value (<NUM>). The first compensated touch value can be generated based on the first measured touch value and a first scaling value. Examples of scaling values are shown in <FIG>. The method can also include generating a second compensated touch value (<NUM>). The second compensated touch value can be generated based on the second measured touch value and a second scaling value. The second scaling value can be less than the first scaling value. The method can also include processing the first touch input and the second touch input (<NUM>). The first touch input and the second touch input can be processed based on the first compensated touch value and the second compensated touch value.

According to an example, a ratio between the first scaling value and the second scaling value can be inversely proportional to a ratio of the density of the touch sensors in the first location to the density of the touch sensors in the second location.

According to an example, the first measured touch value can indicate a change in capacitance in response to the first touch input.

According to an example, the second measured touch value can indicate a change in capacitance in response to the second touch input.

According to an example, the first measured touch value can be received from a touchscreen, such as the touchscreen <NUM>.

According to an example, the first measured touch value can be received from a capacitive touchscreen.

<FIG> is a flowchart showing a method performed by the computing device according to an example implementation. The method can include determining that a measured location of a touch input on a touchscreen is within a predefined area (<NUM>). The predefined area, such as the predefined area <NUM>, can be proximal to a camera <NUM>. The method can include determining a shift value (<NUM>). The shift value, examples of which are shown graphically in <FIG>, can be for the touch input based on the measured location of the touch input. The method can include determining a shifted location (<NUM>). The shifted location can be determined for the touch input based on the measured location and the shift value. The method can include processing the touch input (<NUM>). The touch input can be processed based on the shifted location.

According to an example, the predefined area can have a shape that is homeomorphic with an annulus, and a hole, such as the hole <NUM>, can be bounded by the shape. The hole can superpose at least a portion of the camera.

According to an example, determining the shift can comprise mapping the location of the touch input to the shift value.

<FIG> is a flowchart showing a method performed by a computing system according to an example implementation. The method can include contacting a touchscreen at multiple known locations (<NUM>). An example of contacting the touchscreen <NUM> at multiple known locations is shown in <FIG>. An example of identifying known locations is shown in <FIG>. The multiple known locations can include at least a first location, such as location <NUM>, with a first density of touch sensors, and a second location, such as location <NUM>, with a second density of touch sensors. The second density of touch sensors can be less than the first density of touch sensors. The method can also include determining multiple measured locations (<NUM>). The multiple measured locations can be on the touchscreen <NUM>. The multiple measured locations can correspond to the multiple known locations. The method can also include generating a map (<NUM>). The map, an example of which can either be the shift map <NUM> shown in <FIG> or a map that substitutes measured locations to known locations, can map the multiple measured locations to the multiple known locations.

According to an example, the method can further comprise storing the map on a memory of a mobile computing device, such as the memory <NUM> of the computing device <NUM>.

According to an example, the touchscreen can a first touchscreen, such as a first touchscreen on a measuring and/or calibrating device. The method further can further include receiving a touch input on a second touchscreen, such as the touchscreen <NUM>. The method can further include determining that a location of the touch input on the second touchscreen <NUM> was within a predefined area, such as the predefined area <NUM>, proximal to a camera <NUM>. The method can further include determining a shift value, such as a shift value shown in <FIG>, for the touch input based on the location of the touch input and the generated map. An example of the generated map can be the shift map <NUM> shown in <FIG>. The method can also include processing the touch input based on the location of the touch input and the determined shift value. The touch input can be processed by a client device, such as the computing device <NUM>.

<FIG> is a flowchart showing a method performed by the computing device <NUM> according to an example implementation. The method can include receiving multiple touch inputs (<NUM>). The multiple touch inputs can be received at different times and can be included in a single moving contact. An example of the multiple touch inputs <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> is shown in <FIG>. The method can include determine that a measured location of at least one of the multiple touch inputs is skewed (<NUM>). In the example of <FIG>, touch inputs <NUM>, <NUM> are determined to be skewed. The method can include, based on determining that the location of at least one of the multiple touch inputs is skewed, correcting the skewed location of the at least one of the multiple touch inputs (<NUM>). The skewed location can be corrected based on the location of at least one of the multiple touch inputs that is skewed and locations of at least two other of the multiple touch inputs. Examples of at least two other of the multiple touch inputs are touch inputs <NUM>, <NUM>, <NUM>, <NUM>. The method can also include processing the multiple touch inputs with the corrected location (<NUM>). The corrected location(s) can include corrected touch locations <NUM>, <NUM>.

According to an example, correcting the skewed location can include applying a filter to locations of the multiple touch inputs.

According to an example, correcting the skewed location can include applying an infinite impulse response filter to locations of the multiple touch inputs.

According to an example, correcting the skewed location can include applying a Kalman filter to locations of the multiple touch inputs.

According to an example, correcting the skewed location can include applying a Butterworth filter to locations of the multiple touch inputs.

According to an example, the multiple touch inputs can be received via a touchscreen, such as the touchscreen <NUM>.

According to an example, the multiple touch inputs can be received via a capacitive touchscreen.

According to an example, the method can further include determining that the multiple touch inputs were received within a predetermined time period. The correcting the skewed location can comprise correcting the skewed location based on determining that the multiple touch inputs were received within the predetermined time period and determining that the location of at least one of the multiple touch inputs is skewed.

<FIG> shows an example of a generic computer device <NUM> and a generic mobile computer device <NUM>, which may be used with the techniques described here. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, tablets, workstations, personal digital assistants, televisions, servers, blade servers, mainframes, and other appropriate computing devices. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices.

The processor <NUM> can be a semiconductor-based processor. The memory <NUM> can be a semiconductor-based memory.

Thus, for example, expansion memory <NUM> may be provided as a security module for device <NUM>, and may be programmed with instructions that permit secure use of device <NUM>.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the scope of the invention provided that they are comprised in the scope of the attached claims.

Claim 1:
A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing device (<NUM>) including a camera (<NUM>) included behind a touchscreen (<NUM>) to:
receive a first measured touch value from the touchscreen (<NUM>), the first measured touch value indicating first touch input at a first location;
receive a second measured touch value from the touchscreen (<NUM>), the second measured touch value indicating second touch input at a second location farther from the camera (<NUM>) than the first location, a density of touch sensors in the second location being greater than a density of touch sensors in the first location;
generate a first compensated touch value based on the first measured touch value and a first scaling value;
generate a second compensated touch value based on the second measured touch value and a second scaling value, the second scaling value being less than the first scaling value; and
process the first touch input and the second touch input based on the first compensated touch value and the second compensated touch value.