DIGITAL PEN FOR BRUSH SIMULATION INPUT

A digital pen for drawing via a display panel has a body and a tip. The body includes processing circuitry for processing input data and communicating input control to a processor associated with the display panel. The tip, which has a connector for coupling to the body and a contact interface, includes capture circuitry that processes sensor data from the contact interface and produces input data interfaced to the processing circuitry. The contact interface is defined by the flexible fibers, each of which has a fiber end configured to contact the display panel for producing sensor data. The sensor data identifies an amount of flex contact of the flexible fibers, and the amount of flex contact is correlated to an intensity of line draw for the fiber ends of the fibers. The input control includes instructions usable by the processor to produce a simulated brush stroke on the display panel.

BACKGROUND

Digital painting is an art form in which traditional painting techniques are applied using digital tools by means of a computer, a graphics tablet and an input device, e.g., a digital pen or stylus, and content creation software. In digital painting, the artist moves the digital pen on a digital canvas, e.g., the surface of a graphics tablet, and this movement of the digital pen is captured as a single input for use by the content creation software. The content creation software generates the virtual brush stroke using the single input from the digital pen. As such, the virtual brush stroke is generated, for the most part, by the content creation software and does not truly reflect the ability of the artist.

It is in this context that embodiments arise.

SUMMARY

In an example embodiment, a digital pen for use in drawing via a display panel is provided. The body of the digital pen, which has a first end and a second end, includes processing circuitry for processing input data and communicating input control to a processor associated with the display panel. The tip of the digital pen has a connector for coupling to the first end of the body of the pen and a contact interface opposite the connector. The tip of the pen also includes capture circuitry that processes sensor data from the contact interface and produces the input data that is interfaced to the processing circuitry of the body. The contact interface is defined by a plurality of flexible fibers, with each of the plurality of flexible fibers having a fiber end that is configured to contact a surface of the display panel for producing the sensor data. The sensor data identifies an amount of flex contact of each of the plurality of flexible fibers, and the amount of flex contact is correlated to an intensity of line draw for each of the fiber ends of the plurality of flexible fibers. The input control includes instructions usable by the processor to produce a simulated brush stroke on the display panel.

In one embodiment, each of the plurality of flexible fibers is defined by a fiber optic strand and an end opposite the fiber end of each fiber optic strand connects to an optical sensor array. In one embodiment, the optical sensor array is configured to individually emit light to each fiber optic strand and receive reflected light from each of the respective fiber ends. The received reflected light represents the sensor data from the contact interface.

In one embodiment, the tip of the digital pen further includes a microcontroller that receives the sensor data and produces the input data that is communicated to the connector via a pen tip interface. In one embodiment, the simulated brush stroke is in part generated by digitally producing a plurality of interpolated line draws with respective line weights based on the intensity line draw determined for each of the plurality of flexible fibers. In one embodiment, the simulated brush stroke produces a graphical brush stroke on an interface rendered on the display panel.

In one embodiment, the simulated brush stroke produces a graphical brush stroke on an interface rendered on the display panel, and the capture circuitry and the processing circuitry of the digital pen are configured to produce the input control for a period of time when at least one of the fiber ends produces sensor data indicative of a continuing of the simulated brush stoke along a drawing trajectory across a portion of the display panel.

In one embodiment, the simulated brush stroke is in part generated by digitally producing a plurality of interpolated line draws with respective line weights based on the intensity of the line draw determined for each of the plurality of flexible fibers, and the intensity of the line draw changes, responsive to an amount of the contact of the plurality of flexible fibers with the surface of the display panel, wherein the changes in the intensity of the line draw affect an intensity of the simulated brush stroke while moving the plurality of flexible fibers along the drawing trajectory.

In one embodiment, each of the plurality of flexible fibers includes a shielding cover that extends toward the fiber ends, the fiber ends being free of the shielding cover to enable the fiber ends to contact the surface of the display panel. In one embodiment, the digital pen further includes a magnetic cover that surrounds the shielding cover. The magnetic cover is configured to assist in keeping the plurality of flexible fibers substantially together.

In one embodiment, the processing circuitry includes a communication chip, a microcontroller, a pen body interface, and a power source. In one embodiment, the capture circuitry includes an optical sensor array, a microcontroller with cache, and a pen tip interface. In one embodiment, the plurality of flexible fibers includes at least three flexible fibers and at least three corresponding fiber ends.

In another example embodiment, a method for generating input control from a digital pen, which has a pen body and a tip and is used for drawing on a display panel, is provided. The method includes capturing sensor data from one or more of a plurality of flexible fibers that extend out of the tip, and detecting an intensity of line draw associated with one or more of the plurality of flexible fibers. The intensity of the line draw is correlated to an amount of contact each of the one or more of the plurality of flexible fibers receives when placed upon a surface of the display panel. The method also includes sending input control to a processor, with the input control including data usable by the processor to generate a simulated brush stroke. The simulated brush stroke is configured to be rendered on the display panel responsive to the amount of contact each of the one or more of the plurality of flexible fibers receives when placed upon the surface of the display panel.

In one embodiment, generating the simulated brush stroke by the processor includes processing an interpolation process, with the interpolation process being configured to digitally produce a plurality of interpolated line draws with respective line weights based on the intensity of the line draw associated with each of one of more of the plurality of flexible fibers.

In one embodiment, each of the plurality of flexible fibers is defined by a fiber optic strand, and capturing sensor data from the plurality of flexible fibers that extend out of the tip includes individually emitting light to each fiber optic strand, and receiving reflected light from ends of one or more of the fiber optic strands, with the ends of the one or more of the fiber optic strands contacting a surface of the display panel.

In one embodiment, the capturing of sensor data from the plurality of flexible fibers that extend out of the tip includes capturing sensor data from at least three flexible fibers. In one embodiment, the plurality of flexible fibers that extend out of the tip is arranged in a pattern that simulates a round brush. In one embodiment, the plurality of flexible fibers that extend out of the tip is arranged in a pattern that simulates a flat brush. In one embodiment, the plurality of flexible fibers that extend out of the tip is arranged in a pattern that simulates a fan brush.

Other aspects and advantages of the disclosures herein will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate by way of example the principles of the disclosures.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the example embodiments. However, it will be apparent to one skilled in the art that the example embodiments may be practiced without some of these specific details. In other instances, process operations and implementation details have not been described in detail, if already well known.

FIG. 1illustrates a simplified overview of a system for digital painting including a digital pen having multiple inputs, in accordance with one embodiment. As shown inFIG. 1, system100includes a digital pen102, a display panel104, and a processor106associated with the display panel. The processor106can be located on the display panel104, e.g., the display panel is part of a tablet computer, or can be located on a computer, e.g., a desktop or laptop computer, connected to the display panel. The digital pen102includes a pen body102aand a pen tip102b,which has multiple inputs102b-1. As shown inFIG. 1, pen tip102bhas three inputs; however, the number of inputs can be greater than three as will be described in more detail below.

In use, a user, e.g., a digital artist, moves the digital pen102on the surface104aof the display panel104so that the inputs102b-1contact the surface of the display panel. As the digital pen102moves on the surface104aof the display panel104, the inputs102b-1produce sensor data that reflects the relative amount of contact that the inputs have with the surface of the display panel. The digital pen102also can include an inertial sensor that produces sensor data that reflects the angle of the digital pen and the motion, e.g., acceleration, of the digital pen. In one embodiment, the relative amount of contact that each of the inputs102b-1has with the surface104aof the display panel104is correlated to the intensity of line draw for each of the inputs. The sensor data is interfaced to processing circuitry of the pen body102aand the processing circuitry communicates input control to the processor106associated with the display panel104. The processor106executes content creation software to generate a simulated brush stroke and the simulated brush stroke produces a graphical brush stroke on an interface rendered on the display panel104.

FIG. 2is a simplified schematic diagram that illustrates additional details of a digital pen having multiple inputs, in accordance with one embodiment. As shown inFIG. 2, digital pen102includes pen body102aand pen tip102b.The pen body102aincludes processing circuitry102c,which includes pen body interface108, microcontroller110, power source112, and communication chip114. Power source112, e.g., a rechargeable battery, is coupled to each of pen body interface108, microcontroller110, and communication chip114to provide power thereto. Power source112also provides power for the electronic components in the pen tip102b.The processing circuitry102cprocesses input data and communicates input control116to processor106associated with display panel104, as will be explained in more detail below.

Pen tip102bincludes capture circuitry102d,which includes sensor array118, microcontroller120provided with cache122, and pen tip interface124. A plurality of flexible fibers126extends from sensor array118. One end of each flexible fiber126is connected to sensor array118and the other end of each flexible fiber forms part of contact interface128of the pen tip102b.In particular, the contact interface128is defined by the plurality of flexible fibers126, with each of the flexible fibers having a fiber end126ethat is configured to contact the surface of a display panel, e.g., display panel104, for producing sensor data. In one embodiment, each of the plurality of flexible fibers126is defined by a fiber optic strand. In this embodiment, the sensor array118is an optical sensor array and the end opposite of the fiber end126eof each fiber optic strand126connects to the optical sensor array. The capture circuitry102dprocesses sensor data from the contact interface128and produces the input data that is interfaced to the processing circuitry102cof the pen body102a,as will be explained in more detail below.

The pen body102aand the pen tip102bcan be joined together by any suitable connection technique. As shown inFIG. 2, pen body102aincludes connector130aand pen tip102bincludes connector130b.The connectors130aand130bcan be any suitable connectors, e.g., snap-fit connectors or threaded connectors. In one embodiment, connector130bis a threaded connector including a threaded insertion portion and connector130ais a threaded connector including a complementarily threaded receiving portion. The connectors130aand130balso include corresponding pins and receptacles. As such, in this example embodiment, when the connectors130aand130bare threaded together to create a secure mechanical connection between the pen body102aand the pen tip102b,the corresponding pins and receptacles in the connectors are brought into electrical contact with one another to allow sensor data to be transmitted from the pen tip to the pen body. The pins and receptacles in the connectors130aand130balso can be used to transmit power from the power source112in the pen body102ato the electronic components in the pen tip102bincluding pen tip interface124, microcontroller120provided with cache122, and sensor array118. Alternatively, power can be transmitted from the pen body102ato the pen tip102balong the outside of the threaded insertion portion of connector130b.

In the example embodiment in which sensor array118is an optical sensor array, the optical sensor array includes a light emitter, e.g., a light emitting diode (LED), and a sensor for each of the fiber optic strands126. The light from a light emitter travels along a fiber optic strand126until it reaches the fiber end126eof the fiber optic strand. In the event the fiber end126eis in contact with a reflective surface, e.g., the surface of a display panel, the light reflected from the reflective surface will travel back along the fiber optic strand126and get detected as reflected light by the sensor associated with the fiber optic strand. Those skilled in the art will appreciate that the amount of reflected light detected by the sensor can include a certain amount of noise resulting from, e.g., light reflecting from other fiber optic strands, increased reflection caused by an extremely reflective surface, etc. As such, the sensor data used by the content creation software should be normalized to account for such noise.

To enable the reflected light to be detected by the sensor without interference from light being emitted by the light emitter, the light emitter does not emit light continuously, but rather emits light in pulses. In other words, the light emitter alternates between being on for a certain period of time, during which light is emitted, and being off for a certain period of time, during which no light is emitted. The sensor detects the reflected light during the period of time in which the light emitter is off. In one embodiment, the period of time is about a tenth of a second. Those skilled in the art will appreciate that the rate at which the light is pulsed may be varied to meet the needs of particular applications.

In this example embodiment, the microcontroller120controls the optical sensor array118. In particular, the microcontroller120controls the pulsing of the light emitters associated with each of the fiber optic strands126, e.g., the microcontroller controls when the light emitters are on and when the light emitters are off. The microcontroller120also controls when the sensors associated with each of the fiber optic strands126capture the reflected light and send this sensor data to cache122. In this manner, the microcontroller120ensures that the sensor data is stored in an ordered fashion in the cache122. In particular, at each point in time at which sensor data is captured, e.g., t0, t1, t2, t3, . . . tn, the sensor data for each of the sensors, e.g., S1, S2, S3, . . . Sn, at that point in time is stored in the cache122. Thus, in an example embodiment in which the optical sensor array118includes12sensors, the sensor data for each of the12sensors captured at a particular time, e.g., t0, would be stored in the cache122in association with time t0. The next time sensor data is captured, e.g., time t1, the sensor data captured for each of the12sensors at time t1would be stored in cache122in association with time t1. By storing the sensor data in the cache122in this manner, the cache enables the sensor data for each of the sensors to be ascertained over a period of time, e.g., the period of time during which a digital pen is moved on the surface of a display panel.

The pen tip interface124provides a path for the microcontroller120to send the sensor data in the cache122to the pen body102a.By way of example, the pen tip interface124can be any suitable port, e.g., a serial port or a parallel port. The microcontroller120arranges the sensor data for transmission based on the type of port, as is well known by those skilled in the art, and the pen tip interface124transmits the sensor data to the pen body102aas input data. The pen body interface108of the pen body102areceives the input data and, if needed, the microcontroller110of the pen body rearranges the input data. The microcontroller110sends the input data to communication chip114, which will reformat the input data in accordance with the communication protocol the communication chip is using. In one embodiment, the communication chip114uses the Bluetooth Low Energy (BLE) protocol. Those skilled in the art will appreciate that the communication chip114can also use other suitable wireless protocols.

Using the BLE protocol or other wireless protocol, the communication chip114transmits input control116to a paired device with which processor106is associated. In one embodiment, the paired device is a tablet computer and the processor106is located on the display panel104of the tablet computer. In another embodiment, the paired device is a computer and the processor106is located on the computer, which is connected to the display panel104. The input control116includes the data generated on the digital pen102, e.g., the sensor data from the optical sensor array118which is stored in the cache122as ordered data and then sent to the pen body102aas input data. The content creation software, e.g., an application program, that resides on the processor106uses the input control116to create simulated brush strokes, as will be described in more detail below. The simulated brush strokes produce graphical brush strokes on an interface rendered on the display panel104.

Settings132can be provided to processor106based on input received from the user via the application program. By way of example, the user can input settings132via a menu, a graphical slider, or other suitable interface of the application program. The settings132can include preferences for aspects of the digital painting process. In one embodiment, the settings132include preferences regarding the sensitivity of the digital pen to the user's input. For example, by adjusting the settings, the user could make the digital pen either more or less sensitive to pressure applied by the user. In one embodiment, the settings132include preferences regarding the graphical output generated using the ordered data from the digital pen. For example, by adjusting the settings, the user could adjust characteristics of the graphical brush strokes produced on the interface rendered on the display panel including, e.g., line density, color intensity, etc.

FIG. 3is a simplified diagram that illustrates additional details of the pen tip, in accordance with one embodiment. As shown inFIG. 3, optical sensor array118includes a sensor136and a light emitter134for each fiber optic strand126. In one embodiment, the light emitter134is a light emitting diode (LED). As described above with reference toFIG. 2, the light emitter134, responsive to control signals from microcontroller120, emits pulses of light which travel through fiber optic strand126to fiber end126e.In the event fiber end126eis in contact with a reflective surface, e.g., the surface of a display panel, the light reflected from the reflective surface will travel back along the fiber optic strand126and get detected as reflected light by the sensor136associated with the fiber optic strand. The sensor data from sensors136is stored in cache122in association with the time at which the sensor data was captured, as described above with reference toFIG. 2.

As shown inFIG. 3, each fiber optic strand126is surrounded by a sleeve138. The sleeve138is configured such that the end portion of each fiber optic strand126protrudes from the sleeve. This configuration ensures that the fiber end126eof each fiber optic strand126can directly contact the surface of a display panel or other reflective surface. In one embodiment, the sleeve138includes both an inner portion and an outer portion. In one embodiment, the inner portion of sleeve138is a shielding cover comprised of a metallic material, e.g., a thin aluminum foil, and the outer portion of the sleeve is a magnetic cover (or jacket) comprised of an insulating plastic material, e.g., polyvinyl chloride (PVC), having a magnetic material dispersed therein. In one embodiment, the insulating plastic material is a composite material that includes an amount of a ferromagnetic material, e.g., iron, nickel, etc., dispersed therein. The amount of the ferromagnetic material dispersed within the insulating plastic material is sufficient to render the composite material magnetic. As such, depending on the polarity, the sleeves made of the composite material will magnetically attract to one another and clump together when pressure is applied to the fiber optic strands during use of the digital pen. In this manner, the magnetic cover that forms part of sleeve138assists in keeping the plurality of flexible fibers substantially together during use of the digital pen.

FIG. 4Ais a sectional view illustrating additional details regarding the configuration of the pen tip and the connection of the pen tip to the pen body, in accordance with one embodiment. As shown inFIG. 4A, a plurality of fiber optic strands126protrude from the pen tip102b.Each of the fiber optic strands126is surrounded by a sleeve138such that the end portion of each fiber optic strand, including fiber end126e,protrudes from the sleeve. Connector130bof pen tip102bincludes threaded portion130b-1and pin130b-2. Connector130aof pen body102aincludes threaded portion130a-1, which includes complementary threads for receiving threaded portion130b-1, and receptacle130a-2. To connect pen tip102band pen body102a,threaded portions130b-1and130a-1are threaded together to create a secure mechanical connection between the pen tip and the pen body. In the course of threading the threaded portions130b-1and130a-1together, the pin130b-2and the receptacle130a-2are brought into electrical contact with one another. As described above with reference toFIG. 2, the connection between the pin130b-2and the receptacle130a-2allows sensor data and power to be transmitted from the pen tip102bto the pen body102a.Those skilled in the art will appreciate that more than one pin can be provided on the pen tip, and that the number of receptacles provided on the pen body will correspond to the number of pins provided on the pen tip.

FIG. 4Bis a cross-sectional view of the pen tip taken along section A-A shown inFIG. 4A, in accordance with one embodiment. As shown inFIG. 4B, the plurality of fiber optic strands126in the illustrated central slice of the pen tip102bare arranged in a line. In one embodiment, as can be seen inFIG. 4A, the fiber ends126eof these fiber optic strands126define a curved surface wherein, relative to a fixed reference plane on the pen tip102b,the fiber ends of the fiber optic strands at the center of the pen tip extend further in a longitudinal direction than do the fiber ends of the fiber optic strands at the periphery of the pen tip. With this configuration, if the pen tip102bwere to be oriented perpendicularly to the surface of a display panel, the fiber ends of the fiber strands at the center of the pen tip would be in contact the surface of the display panel, whereas the fiber ends of the fiber optic strands at the periphery of the pen tip would not be in contact with the surface of the display panel. In another embodiment, the fibers ends126eof these fiber optic strands126define a straight line wherein, relative to a fixed reference plane on the pen tip102b,the fiber ends of the fiber optic strands at the center of the pen tip extend the same distance in a longitudinal direction as do the fiber ends of the fiber optic strands at the periphery of the pen tip. With this configuration, if the pen tip102bwere to be oriented perpendicularly to the surface of a display panel, the fiber ends of each the fiber strands would be in contact the surface of the display panel.

FIGS. 5A and 5Bare front and side views, respectively, of the pen tip that shows the location of the holes formed in the pen tip for the flexible fibers, in accordance with one embodiment. As shown inFIG. 5A, the pen tip102bhas 5 holes102b-hformed therein. As can be seen in bothFIGS. 5A and 5B, one of the holes102b-his centrally located on the pen tip102band the other holes are peripherally located around the pen tip. When the pen tip is configured for a relatively low number of flexible fibers, e.g., 3 to 5 flexible fibers, the flexible fibers can be relatively thick. In one embodiment, the flexible fibers have a diameter of approximately 1 millimeter. Each of the holes102b-hshould have a size sufficiently large to accommodate the combined size, e.g., diameter, of the flexible fiber and any sleeve surrounding the flexible fiber.

FIG. 6is a front view of the pen tip that shows the location of the holes formed in the pen tip for the flexible fibers, in accordance with another embodiment. As shown inFIG. 6, the pen tip102bhas 17 holes102b-hformed therein. As can be seen inFIG. 6, one of the holes102b-h3is centrally located on the pen tip102band the other holes are located around the pen tip. In particular, a first group of 8 holes102b-h1is disposed around central hole102b-h3at a first radial distance from the central hole. A second group of 8 holes is peripherally disposed around the pen tip102bat a second radial distance from the central hole102b-h3, with the second radial distance being longer than the first radial distance. When the pen tip is configured for a mid-range number of flexible fibers, e.g., 15 to 30 flexible fibers, the flexible fibers should be relatively thin. In one embodiment, the flexible fibers have a diameter of approximately 0.1 millimeter. Each of the holes102b-h1,102b-h2, and102b-h3should have a size sufficiently large to accommodate the combined size, e.g., diameter, of the flexible fiber and any sleeve surrounding the flexible fiber.

FIG. 7is a front view of the pen tip that shows the location of the holes formed in the pen tip for the flexible fibers, in accordance with yet another embodiment. As shown inFIG. 7, the pen tip102bhas 61 holes formed therein. As can be seen inFIG. 7, one of the holes102b-h3is centrally located on the pen tip102band the other 60 holes are located around the pen tip. In particular, the other holes include six groups of holes102b-h4,102b-h5,102b-h6,102b-h7,102b-h8, and102b-h9. Each of these six groups of holes includes a total of 11 holes, which extend in linear fashion from one side of the pen tip102bto the other side of the pen tip and pass through the central hole102b-h3. As such, central hole102b-h3is common to each of the six groups of holes. In one embodiment, the lines of holes are disposed around the pen tip102bat approximately 30 degree intervals. Considering, for example, the group of holes102b-h4, there are 5 holes102b-h4on the left side of the central hole102b-h3and 5 holes102b-h4on the right side of the central hole. Thus, including the central hole102b-h3, the line of holes102b-h4includes a total of 11 holes. When the pen tip is configured for a relatively high number of flexible fibers, e.g., 50 to 100 flexible fibers, the flexible fibers should be extremely thin. In one embodiment, the flexible fibers have a diameter of in a range from approximately 0.04 millimeter to approximately 0.06 millimeter. Each of the holes102b-h3,102b-h4,102b-h5,102b-h6,102b-h7,102b-h8, and102b-h9should have a size sufficiently large to accommodate the combined size, e.g., diameter, of the flexible fiber and any sleeve surrounding the flexible fiber.

FIG. 8is a front view of the pen tip that shows the location of the holes formed in the pen tip for the flexible fibers, in accordance with a further embodiment. As shown inFIG. 8, the pen tip102bhas 97 holes formed therein. As can be seen inFIG. 8, one of the holes102b-h3is centrally located on the pen tip102band the other 96 holes are located around the pen tip. In particular, the other holes include the six groups of holes102b-h4,102b-h5,102b-h6,102b-h7,102b-h8, and102b-h9described above with reference toFIG. 7(these six groups include a total of 60 holes). In addition, the other holes include twelve groups of holes102b-h10, with each of these groups including 3 holes (thus these twelve groups include a total of 36 holes). Each of the twelve groups of holes102b-h10is located between two of the six groups of holes102b-h4,102b-h5,102b-h6,102b-h7,102b-h8, and102b-h9. For example, three holes102b-h10on the left side of the pen tip102bare located between the groups of holes102b-h4and102b-h5, and three holes102b-h10on the right side of the pen tip are located between the groups of holes102b-h4and102b-h9. As the pen tip102bin the example embodiment ofFIG. 8is configured for a relatively high number of flexible fibers, e.g., 50 to 100 flexible fibers, the flexible fibers should be extremely thin, e.g., approximately 0.04 millimeter to approximately 0.06 millimeter. Each of the holes102b-h3,102b-h4,102b-h5,102b-h6,102b-h7,102b-h8,102b-h9, and102b-h10should have a size sufficiently large to accommodate the combined size, e.g., diameter, of the flexible fiber and any sleeve surrounding the flexible fiber.

As the number of flexible fibers, e.g., fiber optic strands, protruding from the pen tip increases, the likelihood of interference from light traveling through neighboring fiber optic strands increases. To reduce the likelihood of such interference, the distance by which each fiber optic strand extends beyond the end of the sleeve surrounding the fiber optic strand should be reduced as the density of fiber optic strands on the pen tip increases.FIG. 9shows a sleeve surrounding a fiber optic strand such that an end portion of the fiber optic strand extends beyond the end of the sleeve, in accordance with one embodiment. As shown inFIG. 9, sleeve138surrounds fiber optic strand126and an end portion of the fiber optic strand extends beyond the end of the sleeve. The distance X, which extends from the end of the sleeve138to fiber end126eof fiber optic strand126, defines the distance by which the fiber optic strand extends beyond the end of the sleeve. In one example embodiment, the distance X is in a range from approximately 0.25 millimeter to approximately 5 millimeters. In another example embodiment, the distance X is in a range from approximately 0.5 millimeter to approximately 2 millimeters. In yet another example embodiment, the distance X is approximately 1 millimeter. Reducing the distance by which the fiber optic strand extends beyond the end of the sleeve (the distance X) as the density of the fiber optic strands on the pen tip increases helps ensure that the substantial majority of light detected by the sensor associated with each fiber optic strand is light reflected from the surface of the display panel, rather than light from neighboring fiber optic strands. In the example embodiments in which the distance X is relatively short, e.g., in the range from 0.25 millimeter to 2 millimeters, the degree to which the exposed portion of the fiber optic strand can flex or bend when pressed against the surface of the display panel might be limited, but the flexibility of the sleeve will allow the fiber optic strand as a whole to flex or bend so that the amount of light reflected from the surface of the display panel will be indicative of the amount of pressure being applied to the surface of the display panel by the fiber optic strand.

FIG. 10Aillustrates a digital pen having multiple inputs being moved on the surface of a display panel, in accordance with one embodiment. As shown inFIG. 10A, digital pen102is being moved on the surface104aof display104along a drawing trajectory from point A to point B. In this example embodiment, digital pen102includes four flexible fibers, e.g., fiber optic strands,126-1,126-2,126-3, and126-4, which protrude from pen tip102b(for ease of illustration, the flexible fibers126-1,126-2,126-3, and126-4are shown without the sleeves that would be present in normal use). As can be seen inFIG. 10A, as the digital pen102is being moved from point A to point B, the fiber ends126e-1,126e-2, and126e-3are being pressed into contact with the surface104aof the display panel104, whereas fiber end126e-4is not in contact with the surface of the display panel. The arrows inFIG. 10Aindicate that digital pen102is simultaneously being pressed against the surface104aof the display panel104as well as being moved on the surface of the display panel from point A to point B. The intensity of the line draw for each of the flexible fibers depends on the amount of flex contact for each of the fiber ends of the flexible fibers, as will be explained in more detail below with reference toFIG. 10B.

FIG. 10Billustrates the line draw for each the flexible fibers shown inFIG. 10A, in accordance with one embodiment. As shown inFIG. 10B, there are three line draws L1, L2, and L3on the surface104aof display panel104, and these line draws extend substantially from point A to point B. Line draw L1is the result of the contact of fiber end126e-1with the surface104aof the display panel104. Line draw L2is the result of the contact of fiber end126e-2with the surface104aof the display panel104. Line draw L3is the result of the contact of fiber end126e-3with the surface104aof the display panel104. There is no line draw corresponding to fiber end126e-4inFIG. 10Bbecause, as shown inFIG. 10A, fiber end126e-4was not in contact with the surface104aof the display panel104as the digital pen102was being moved from point A to point B. The line draws L1, L2, and L3shown inFIG. 10Bare the “raw” line draws resulting from the movement of the digital pen102on the surface104aof the display panel104before the line draws have been interpolated, e.g., by content creation software, to generate a simulated brush stroke, which produces a graphical brush stroke on an interface rendered on the display panel. Additional detail regarding the interpolation of the line draws are described below with reference toFIG. 10C.

As can be seen in the enlarged portion ofFIG. 10B(the portion inside the dashed circle), the intensity, e.g., the thickness and boldness, of the line draw L1is greater than the intensity of either line draws L2or L3. The reason for this is that the pressure being applied to fiber optic strand126-1as the digital pen102is being moved from point A to point B is higher than the pressure being applied to either fiber optic strands126-2or126-3. Consequently, as can be seen inFIG. 10A, the amount of flex contact that fiber optic strand126-1has with the surface104aof display panel104is greater than the amount of flex contact that either fiber optic strands126-2or126-3have with the surface of the display panel because the increased pressure has caused the fiber optic strand126-1to flex or bend more than either fiber optic strands126-2or126-3have flexed or bent. As the result of the increased flex contact fiber optic strand126-1has with the surface104aof the display104, the fiber end126e-1of fiber optic strand126-1will reflect relatively more light back to the associated sensor in the optical sensor array and thus cause this sensor to detect a more robust signal.

As also can be seen in the enlarged portion ofFIG. 10B, the intensity, e.g., the thickness and boldness, of the line draw L2is greater than the intensity of the line draw L3. The reason for this is that the pressure being applied to fiber optic strand126-2as the digital pen102is being moved from point A to point B is higher than the pressure being applied to fiber optic strand126-3. Consequently, as can be seen inFIG. 10A, the amount of flex contact that fiber optic strand126-2has with the surface104aof display panel104is slightly greater than the amount of flex contact that fiber optic strand126-3has with the surface of the display panel because the increased pressure has caused the fiber optic strand126-2to flex or bend slightly more than fiber optic strand126-3has flexed or bent. As the result of the increased flex contact fiber optic strand126-2has with the surface104aof the display104, the fiber end126e-2of fiber optic strand126-2will reflect relatively more light back to the associated sensor in the optical sensor array and thus cause this sensor to detect a more robust signal.

FIG. 10Cillustrates a graphical brush stroke on an interface rendered on a display panel, in accordance with one embodiment. As shown inFIG. 10C, graphical brush stroke S1appears on an interface rendered on the display panel104. The graphical brush stroke S1is generated by conducting an interpolation process on the line draws L1, L2, and L3(seeFIG. 10B) to generate a simulated brush stroke. The simulated brush stroke is then used to produce the graphical brush stroke S1on the interface rendered on the display panel104. In one embodiment, the interpolation process is carried out by content creation software and includes the use of a randomizing function. By way of example, in the randomizing function, random numbers are generated and these random numbers are used to select the locations at which interpolated data is added to the original line draws (see, e.g.,FIGS. 11A-11Cfor additional details regarding the interpolation process).

As can be seen in the enlarged portion ofFIG. 10C(the portion inside the dashed circle), the lower portion of graphical brush stroke S1is relatively dark, the upper portion of the graphical brush stroke is relatively light, and the center portion of the graphical brush stroke transitions from relatively dark to relatively light (moving in the direction from the bottom of the graphical brush stroke toward the top of the graphical brush stroke). The content creation software generates the graphical brush stroke S1using, among other things, 1) the line draws L1, L2, and L3(seeFIG. 10B), and 2) the interpolation process that adds data to the original line draws at randomly selected locations. The lower portion of the graphical brush stroke S1is relatively dark because this portion corresponds to the location of line draw L1, which had the most intensity, e.g., thickness and boldness, of the line draws L1, L2, and L3due to the relatively high pressure applied to fiber optic strand126-1. The upper portion of the graphical brush stroke S1is relatively light because this portion corresponds to the location of line draw L3, which had the least intensity of the line draws L1, L2, and L3due to the relatively low pressure applied to fiber optic strand126-3. Through the use of the multiple data points provided by, e.g., line draws L1, L2, and L3, as well the interpolation process including the randomizing function, the graphical brush stroke S1generated by the content creation software more closely replicates a brush stroke made by a real paint brush. In particular, one reason for this is that the use of a digital pen with multiple inputs, e.g., multiple fiber optic strands, allows a digital artist to introduce human variance into the digital painting process, e.g., through the application of different amounts of pressure to one or more of the multiple inputs, and thereby capture more of the true ability of the digital artist.

FIGS. 11A-11Cillustrate additional details of the interpolation process used to convert the line draws into a simulated brush stroke, in accordance with one embodiment.FIG. 11Ashows a simplified illustration of a digital pen102being moved (from left to right) on the surface104aof display panel104. As shown inFIG. 11A, digital pen102includes three flexible fibers, e.g., fiber optic strands,126-1,126-2, and126-3, which protrude from pen tip102b(for ease of illustration, the flexible fibers126-1,126-2, and126-3are shown without the sleeves that would be present in normal use). As the digital pen102is being moved along a drawing trajectory, the fiber ends126e-1,126e-2, and126e-3are being pressed into contact with the surface104aof the display panel104.

FIG. 11Bis a graph that shows the resulting line draws for each of the fiber optic strands as a function of time. As shown inFIG. 11B, line draw LD1is the result of the contact of fiber end126e-1with the surface104aof the display panel104along the drawing trajectory. Line draw LD2is the result of the contact of fiber end126e-2with the surface104aof the display panel104along the drawing trajectory. Line draw LD3is the result of the contact of fiber end126e-3with the surface104aof the display panel104along the drawing trajectory. As can be seen in the graph ofFIG. 11B, line draw LD1extends from just before time t1to time t6. Line draw LD2extends from time t0to time t5, which means that the fiber end126e-2was not in contact with the surface104aof the display panel104after time t5. Line draw LD3extends from just before time t1to time t4, which means that fiber end126e-3was not in contact with the surface104aof the display panel104after time t4.

FIG. 11Cillustrates the addition of interpolated data to the line draws LD1, LD2, and LD3during the period of time from t2to t3shown inFIG. 11B. As shown inFIG. 11C, interpolated data lines12and13have been inserted between line draws LD1and LD2. The content creation software determines the intensity, e.g., the thickness and the boldness, of the interpolated data lines12and13to be added between line draws LD1and LD2. The intensity determination for the interpolated data lines is influenced by the intensity of the surrounding line draws LD1and LD2. Thus, by way of example, the interpolated data lines will be thicker and bolder when the line draws LD1and LD2are thicker and bolder. In one embodiment, the positions at which the interpolated data lines12and13are added between line draws LD1and LD2are determined using a randomizing function. In one example embodiment, a random seed is used to initialize a pseudorandom number generator and generate a sequence of random numbers. In this example embodiment, the content creation software uses the random numbers to determine random positions at which the interpolated data lines12and13are added between the line draws LD1and LD2. By randomly inserting the interpolated data between the line draws rather than, e.g., inserting the interpolated data at predefined locations between the line draws, the resulting simulated brush stroke has an appearance that more closely replicates a brush stroke made by a real paint brush.

As shown inFIG. 11C, interpolated data line13has been added between line draws LD2and LD3. As noted above, the intensity, e.g., the thickness and boldness, of the interpolated data line13, which is determined by the content creation software, is influenced by the intensity of the surrounding line draws LD2and LD3. Further, in one example embodiment, the location of interpolated data line13is randomly determined by the content creation software using a randomizing function. As can be seen inFIG. 11C, the random position at which interpolated data line13is located is closer to line draw LD3than to line draw LD2. As noted above, with the use of the randomizing function, the resulting simulated brush stroke has an appearance that more closely replicates a brush stroke made by a real paint brush. Without the randomizing function, an interpolated data line would typically be automatically located at the midway point between line draws, e.g., halfway between line draw LD2and line draw LD3, and therefore would have a less realistic appearance. In the example embodiments shown inFIG. 11C, one or two interpolated data lines have been added between the line draws resulting from the use of the digital pen with multiple inputs. Those skilled in the art will appreciate that any number of interpolated data lines, e.g., 3, 5, 10, 50, etc., can be added in between the original line draws. Those skilled in the art also will appreciate that the randomizing function can be implemented during more than one time period during which the line draws are being generated. By way of example, the randomizing function could be implemented for each time period, e.g., from t2to t3, from t3to t4, from t4to t5, etc., for every other time period, e.g., from t1to t2, from t3to t4, from t5to t6, etc, or for multiple time periods, e.g., from t0to t2, from t2to t4, from t4to t6, etc.

FIG. 12is a flow diagram illustrating the method operations performed in generating input control from a digital pen used for drawing on a display panel, in accordance with an example embodiment. The method begins in operation400in which sensor data is captured from one or more of a plurality of flexible fibers that extend of the tip of the digital pen. In one embodiment, the flexible fibers are fiber optic strands through which light is transmitted. It will be appreciated by those skilled in the art, however, that the method can be implemented using other types of flexible fibers, e.g., thin metallic wires, spring tips, etc. In the example in which the flexible fibers are fiber optic strands, the sensor data can be captured by emitting light to the fiber optic strands, and receiving the reflected light from one or more of the fiber ends of the fiber optic strands contacting the surface of the display panel. The reflected light can be captured by any suitable sensor, e.g., sensor136which forms part of optical sensor array118(see, e.g.,FIG. 3).

In operation402, the intensity of line draw associated with one or more of the plurality of flexible fibers is detected, with the intensity of the line draw being correlated to an amount of contact each of the one or more of the plurality of flexible fibers receives when placed upon the surface of the display panel. The sensor data captured in operation400is raw signal data. In the detecting performed as part of operation402, a basic analysis of this raw signal data is performed, e.g., by a processor, to generate relative magnitudes of intensity of the captured sensor data. As such, the detecting performed as part of operation402will identify the flexible fibers that have had the most flex contact with the surface of the display panel (and therefore generated more reflected light relative to the other flexible fibers) while the digital pen was being moved on the surface of the display panel.

In operation404, input control is sent to a processor, with the input control including data usable by the processor to generate a simulated brush stroke. The simulated brush stroke is configured to be rendered on the display panel responsive to the amount of contact each of the one or more of the plurality of flexible fibers receives when placed upon the surface of the display panel. The input control can be sent to the processor using any suitable wireless protocol. In one embodiment, the input control is sent to a paired device using the Bluetooth Low Energy (BLE) protocol. In one embodiment, the paired device is a tablet computer and the processor is located on the display panel of the tablet computer. In another embodiment, the paired device is a computer and the processor is located on the computer, which is connected to a display panel. In the example embodiment using the BLE protocol, the input control is a packet stream being sent to the processor, with the packet stream including, among other data items, the intensity data generated in operation402. The processor uses the input control, in conjunction with content creation software, e.g., an application program, that resides on the processor create simulated brush strokes that include characteristics based, at least in part, on the intensity data included in the input control.

In one embodiment, the generating of the simulated brush stroke by the processor includes processing an interpolation process. The interpolation process is configured to digitally produce a plurality of interpolated line draws with respective line weights based on the intensity of the line draw associated with each of the one or more of the plurality of flexible fibers. As described above with reference toFIGS. 11A-11C, interpolated data lines are added between the original line draws resulting from use of the digital pen with multiple inputs, e.g., flexible fibers. In addition, interpolated data lines can be added outside of an original line draw, e.g., when the original line draw is the outermost line draw resulting from use of the digital pen with multiple inputs. In one embodiment, the location of the interpolated data lines is determined using a randomizing function. The content creation software determines the intensity, e.g., the thickness and the boldness, of the interpolated data lines to be added between the original line draws (or outside of one of the outermost original line draws). The intensity determination for the interpolated data lines is influenced by the intensity, e.g., the thickness and boldness, of the surrounding line draws (or the most proximate line draw). Thus, by way of example, the interpolated data lines will have greater line weights when the intensity of the surrounding line draws is higher. As used herein, the phrase “line weight” refers to a thickness of a data line in terms of the number of pixels used to render a graphical version of the data line on a display panel. Those skilled in the art will appreciate that the pixel density will vary based on resolution of the tablet display panel. As such, more pixels will be used to render the graphical version of the data line at higher resolutions than will be used to render the graphical version of the data line at lower resolutions.

In one embodiment in which each of the plurality of flexible fibers is defined by a fiber optic strand, the capturing of sensor data from the plurality of flexible fibers that extend out the tip of the digital pen includes individually emitting light to each fiber optic strand, and receiving reflected light from the ends of one or more of the fiber optic strands, with the ends of the one or more fiber optic strands contacting the surface of the display panel. Light can be individually emitted to each fiber optic strand using any suitable light emitter, e.g., a light emitting diode (LED). By way of example, the optical sensor array118shown in, e.g.,FIG. 3, includes a light emitter134for each fiber optic strand126. The reflected light from the ends of one or more fiber optic strands can be received using any suitable sensor. By way of example, the optical sensor array118shown in, e.g.,FIG. 3, includes a sensor136for each fiber optic strand126. As described above, when the fiber end of one of the fiber optic strands is in contact with a reflective surface, e.g., the surface of the display panel, the light reflected from the reflective surface will travel back along the fiber optic strand and get detected as reflected light by the sensor associated with the fiber optic strand.

In traditional painting, an artist uses a variety of brushes to achieve different effects. By way of non-limiting example, these brushes include round brushes, flat brushes, and fan brushes. It will be apparent to those skilled in the art that the flexible fibers, e.g., fiber optic strands, described herein can be arranged to define a shape that is comparable to that of a round brush, a flat brush, or a fan brush used in traditional painting. Those skilled in the art will also appreciate that other techniques can be used to adjust the shape of the flexible fibers protruding from the pen tip of the digital pen. For example, a conformable clamp can be placed around the flexible fibers. By changing the shape of the conformable clamp, e.g., by squeezing the clamp together, the shape of the flexible fibers can be adjusted to a desired shape. Alternatively, the digital pen can be provided with one or more dials that can be used to adjust the length by which each of the flexible fibers extends beyond the outer surface of the pen tip. For example, the digital pen can be provided with a first dial to adjust the overall length of the flexible fibers and a second dial to adjust the radial falloff of the flexible fibers (the degree to which the outer flexible fibers are shorter than the inner flexible fibers).

In the example embodiments described herein, the flexible fibers are fiber optic strands which transmit light that gets reflected from the surface of a display panel. It will be apparent to those skilled in the art that the principles described herein can be implemented using flexible fibers other than fiber optic strands. By way of example, the flexible fibers can be spring tips or thin metallic wires.

As described above, the digital pen transmits input control to a paired device with which a processor is associated. In one example in which the paired device is a tablet computer, the processor can be located on the display panel of the tablet computer. In another example in which the paired device is a computer, the processor can be located on the computer, which is connected to the display panel. In either of these examples, the processing of the data from the digital pen can be carried out in the cloud. To enable such cloud processing of the data, the paired device, e.g., the tablet computer or the computer connected to the display panel, can be connected to the cloud via any suitable internet connection.FIG. 13illustrates an embodiment of an Information Service Provider architecture. Information Service Provider (ISP)970delivers a multitude of information services to users982geographically dispersed and connected via network986. An ISP can deliver just one type of service, such as stock price updates, or a variety of services such as broadcast media, news, sports, gaming, etc. Additionally, the services offered by each ISP are dynamic, that is, services can be added or taken away at any point in time. Thus, the ISP providing a particular type of service to a particular individual can change over time. For example, a user may be served by an ISP in near proximity to the user while the user is in her home town, and the user may be served by a different ISP when the user travels to a different city. The home-town ISP will transfer the required information and data to the new ISP, such that the user information “follows” the user to the new city making the data closer to the user and easier to access. In another embodiment, a master-server relationship may be established between a master ISP, which manages the information for the user, and a server ISP that interfaces directly with the user under control from the master ISP. In another embodiment, the data is transferred from one ISP to another ISP as the client moves around the world to make the ISP in better position to service the user be the one that delivers these services.

ISP970includes Application Service Provider (ASP)972, which provides computer-based services to customers over a network (e.g., including by way of example without limitation, any wired or wireless network, LAN, WAN, WiFi, broadband, cable, fiber optic, satellite, cellular (e.g. 4G, 5G, etc.), the Internet, etc.). Software offered using an ASP model is also sometimes called on-demand software or software as a service (SaaS). A simple form of providing access to a particular application program (such as customer relationship management) is by using a standard protocol such as HTTP. The application software resides on the vendor's system and is accessed by users through a web browser using HTML, by special purpose client software provided by the vendor, or other remote interface such as a thin client.

Services delivered over a wide geographical area often use cloud computing. Cloud computing is a style of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet. Users do not need to be an expert in the technology infrastructure in the “cloud” that supports them. Cloud computing can be divided into different services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services often provide common business applications online that are accessed from a web browser, while the software and data are stored on the servers. The term cloud is used as a metaphor for the internet (e.g., using servers, storage and logic), based on how the internet is depicted in computer network diagrams and is an abstraction for the complex infrastructure it conceals.

Further, ISP970includes a Game Processing Server (GPS)974which is used by game clients to play single and multiplayer video games. Most video games played over the internet operate via a connection to a game server. Typically, games use a dedicated server application that collects data from players and distributes it to other players. This requires a separate server to host the server application. In another embodiment, the GPS establishes communication between the players and their respective game-playing devices exchange information without relying on the centralized GPS. In yet another embodiment, the GPS can be used to distribute game copies to nodes via download and to facilitate a discovery process by which nodes can initiate a peer-to-peer connection with other nodes interested in playing a game in a serverless environment.

Dedicated GPSs are servers which run independently of the client. Such servers are usually run on dedicated hardware located in data centers, providing more bandwidth and dedicated processing power. Dedicated servers are the preferred method of hosting game servers for most PC-based multiplayer games. Massively multiplayer online games run on dedicated servers usually hosted by the software company that owns the game title, allowing them to control and update content.

Broadcast Processing Server (BPS)976distributes audio or video signals to an audience. Broadcasting to a very narrow range of audience is sometimes called narrowcasting. The final leg of broadcast distribution is how the signal gets to the listener or viewer, and it may come over the air as with a radio station or TV station to an antenna and receiver, or may come through cable TV or cable radio (or “wireless cable”) via the station or directly from a network. The internet may also bring either radio or TV to the recipient, especially with multicasting allowing the signal and bandwidth to be shared. Historically, broadcasts have been delimited by a geographic region, such as national broadcasts or regional broadcast. However, with the proliferation of fast internet, broadcasts are not defined by geographies as the content can reach almost any country in the world.

Storage Service Provider (SSP)978provides computer storage space and related management services. SSPs also offer periodic backup and archiving. By offering storage as a service, users can order more storage as required. Another major advantage is that SSPs include backup services and users will not lose all their data if their computers' hard drives fail. Further, a plurality of SSPs can have total or partial copies of the user data, allowing users to access data in an efficient way independently of where the user is located or the device being used to access the data. For example, a user can access personal files in the home computer, as well as in a mobile phone while the user is on the move.

Communications Provider980provides connectivity to the users. One kind of Communications Provider is an Internet Service Provider (ISP) which offers access to the Internet. The ISP connects its customers using a data transmission technology appropriate for delivering Internet Protocol datagrams, such as dial-up, DSL, cable modem, fiber, wireless or dedicated high-speed interconnects. The Communications Provider can also provide messaging services, such as e-mail, instant messaging, and SMS texting. Another type of Communications Provider is the Network Service Provider (NSP) which sells bandwidth or network access by providing direct backbone access to the Internet. Network service providers include telecommunications companies, data carriers, wireless communications providers, Internet service providers, cable television operators offering high-speed internet access, etc.

Data Exchange988interconnects the several modules inside ISP970and connects these modules to users982via network986. Data Exchange988can cover a small area where all the modules of ISP970are in close proximity, or can cover a large geographic area when the different modules are geographically dispersed. For example, Data Exchange988can include a fast Gigabit Ethernet (or faster) within a cabinet of a data center, or an intercontinental virtual area network (VLAN).

Users982access the remote services with client device920, which includes at least a CPU, a memory, a display and I/O. The client device can be a PC, a mobile phone, a netbook, tablet, gaming system, a PDA, etc. In one embodiment, ISP970recognizes the type of device used by the client and adjusts the communication method employed. In other cases, client devices use a standard communications method, such as HTML, to access ISP970.

With the above embodiments in mind, it should be understood that the disclosure can employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the disclosure are useful machine operations. The disclosure also relates to a device or an apparatus for performing these operations. The apparatus can be specially constructed for the required purpose, or the apparatus can be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.

Accordingly, the disclosure of the example embodiments is intended to be illustrative, but not limiting, of the scope of the disclosures, which are set forth in the following claims and their equivalents. Although example embodiments of the disclosures have been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the following claims. In the following claims, elements and/or steps do not imply any particular order of operation, unless explicitly stated in the claims or implicitly required by the disclosure.