Whiteboard imaging system

An image capturing system is installable in a room opposite a writing surface. The image capturing system is adapted to take visual images of the writing surface and identify information written thereon.

BACKGROUND OF THE INVENTION

The present invention relates to devices for obtaining information from a whiteboard or other writing surface. More particularly, the present invention relates to a system for obtaining visual images of the writing surface.

Whiteboards are used commonly in schools, businesses or other gathering locations. Typically, the whiteboard includes a smooth planar surface that is mountable, for example, to a wall or other structure. Pens, typically of different colors, allow one or more individuals to draw on the whiteboard to explain ideas, concepts and the like.

Various systems have been advanced in order to capture and record information written on a whiteboard. For instance, a whiteboard has been designed to include a pressure-sensitive array that senses contact and movement of the pen on the surface. In another embodiment, an ultrasonic sensor is mounted to the whiteboard. The sensor receives ultrasonic signals from transmitters provided on pens or erasers. The sensor triangulates a position of the pen or eraser on the board as the user writes or erases information.

Although the prior art systems have enjoyed some success, widespread adoption of any one technology has not yet been achieved. Significant impediments include the current use of inexpensive whiteboards already installed in many locations. To use some of the systems advanced, these inexpensive whiteboards would have to be discarded and more expensive systems would have to be bought and installed. Although some technologies use current whiteboards, such as the ultrasonic system described above, users must now use special pens, erasers and techniques in order to capture the information.

There thus is a continuing need for a system to capture information on a whiteboard. Such a system would preferably use existing whiteboards, but would not require special pens, erasers, or other techniques that the operator must use or adapt to in order to capture information on the whiteboard.

SUMMARY OF THE INVENTION

An image capturing system is installable in a room opposite a writing surface. The image capturing system is adapted to take visual images of the writing surface and identify information written thereon.

A second broad aspect includes an image capturing system having a visual sensor providing image data corresponding to sensed images. The visual sensor is selectively directed toward a first position to sense a first image and toward a second position to sense a second image. An image processor is coupled to the visual sensor to receive the image data from the visual sensor. The image processor is capable of processing the image data as a function of direction of the visual sensor toward the first position or the second position.

A third broad aspect includes an image capturing system having a visual sensor providing image data corresponding to sensed visual images. The visual sensor includes a set of adjacent sensing elements being exposed collectively to successive portions of the image. In other words, the same sensing elements will be used more than once, to sense different portions, during capture of a single image. A storage device stores sensing element control values. A controller is coupled to the storage device and the visual sensor. The controller controls a time duration of exposure of the sensing elements to the portion of image as a function of exposure to successive portions.

A fourth aspect includes a method of obtaining information provided on a writing surface in a room. The method includes locating an image capturing system at a second location in the room remote from the writing surface; sensing a visual image of the writing surface with the image capturing system; and identifying information provided on the writing surface with the image capturing system.

DETAILED DESCRIPTION OF THE ILLUSTRATIVE EMBODIMENTS

Referring toFIG. 1, a first broad aspect of the present invention includes an image capturing system10disposed in a room11having a defined writing surface14A, such as a whiteboard. In the embodiment illustrated, the image capturing system10includes a visual sensor13that visually senses images provided on the writing surface14A. An image processor20, such as a desktop computer, receives image data from the visual sensor13and is adapted to identify information provided on the writing surface14A.

Generally, the visual sensor13is disposed in a fixed location in the room11so as to obtain substantially identical, successive images of the writing surface14A. InFIG. 1, two exemplary positions are illustrated. In a first position, visual sensor13is mounted to a wall12, or some other like fixed structure, that is generally opposite the writing surface14A so as to face the writing surface14A. In a second position, visual sensor13is disposed on a ceiling17, or other suspended structure, in order to view the writing surface14A from above. The second position is also convenient for obtaining visual images of other writing surfaces, such as writing surface14B. In yet a further embodiment, the visual sensor13can be directed to also obtain visual images of other areas in the room11, for example, a predefined surface area15of a table or desk16located below the visual sensor13. It should be noted that the ceiling location of visual sensor13is not the only location in the room11in order to view multiple writing surfaces or predefined areas. For instance, depending upon the location of the writing surfaces14A and14B in the room11, visual sensor13could be located in a corner or other fixed position in order to selectively view the writing surfaces14A and14B.

FIG.2and the related discussion are intended to provide a brief, general description of a suitable computing environment in which the image processing system10may be implemented. Although not required, the image processing system10will be described, at least in part, in the general context of computer-executable instructions, such as program modules being executed by a controller, a personal computer or other computing device. Generally, program modules include routine programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Those skilled in the art can implement the description, block diagrams and flow charts to computer-executable instructions. In addition, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. The invention is also applicable in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

With reference toFIG. 2, an exemplary environment for the image processor includes a general purpose computing device in the form of a conventional personal computer20, including processing unit21, a system memory22, and a system bus23that couples various system components including the system memory to the processing unit21. The system bus23may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory includes read only memory (ROM)24and random access memory (RAM)25. A basic input/output system26(BIOS), containing the basic routine that helps to transfer information between elements within the personal computer20, such as during start-up, is stored in ROM24. The personal computer20further includes a hard disk drive27for reading from and writing to a hard disk (not shown), a magnetic disk drive28for reading from or writing to removable magnetic disk29, and an optical disk drive30for reading from or writing to a removable optical disk31, such as a CD ROM or other optical media. The hard disk drive27, magnetic disk drive28, and optical disk drive30are connected to the system bus23by a hard disk drive interface32, magnetic disk drive interface33, and an optical drive interface34, respectively. The drives and the associated computer-readable media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for the personal computer20.

Although the exemplary environment described herein employs a hard disk, a removable magnetic disk29and a removable optical disk31, it should be appreciated by those skilled in the art that other types of computer readable media which can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read only memory (ROM), and the like, may also be used in the exemplary operating environment.

A number of program modules may be stored on the hard disk, magnetic disk29, optical disk31, ROM24or RAM25, including an operating system35, one or more application programs36, other program modules37, and program data38. A user may enter commands and information into the personal computer20through input devices such as a keyboard40and pointing device (mouse)42. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices, including the visual sensor13, are often connected to the processing unit21through a serial port interface46that is coupled to the system bus23, but may be connected by other interfaces, such as a sound card, a parallel port, a game port or a universal serial bus (USB). A monitor47or other type of display device is also connected to the system bus23via an interface, such as a video adapter48. In addition to the monitor47, personal computers may typically include other peripheral output devices such as a speaker and printers (not shown).

The personal computer20may operate in a networked environment using logic connections to one or more remote computers, such as a remote computer49. The remote computer49may be another personal computer, a server, a router, a network PC, a peer device or other network node, and typically includes many or all of the elements described above relative to the personal computer20, although only a memory storage device50has been illustrated in FIG.2. The logic connections depicted inFIG. 2include a local are network (LAN)51and a wide area network (WAN)52. Such networking environments are commonplace in offices, enterprise-wide computer network intranets and the Internet.

When used in a LAN networking environment, the personal computer20is connected to the local area network51through a network interface or adapter53.

Although described above wherein a desktop computer20will process image data from the visual sensor13, it should be noted that the visual sensor13could also include some of the components used for processing and storage. For instance, the visual sensor13could include a processor for processing the image data for storage on a mass storage device such as a hard disk or floppy disk. If stored on a floppy disk, the image data can then be removed and transferred to another computer. Likewise, a wired or wireless communications interface can be included to transfer data from the visual sensor13to other computing devices.

FIG. 3is a block diagram of an exemplary form of the visual sensor13. Generally, the visual sensor13includes a visual sensing device60operably coupled to a controller64. As appreciated by those skilled in the art, the visual sensing device60can take many forms, for instance, sensing devices commonly used in digital cameras, video cameras, scanners, fax machines, or digital copiers can be used. Referring to the embodiment illustrated inFIG. 4, the visual sensing device60comprises a linear array of sensing elements (e.g. charge coupled devices “CCD”) such as commonly found in scanners. In order to obtain a complete visual image of the writing surface14A, the linear array60scans the writing surface14A and receives successive portions of the writing surface14A. As is well known, an output signal from the linear array60is related to each successive portion of the writing surface14A. The output signals are provided to controller64. Controller64operates the linear array60. As used herein, the controller64can include components such as a processor, memory and software for operation and an interface for communication with the computer20. If desired, the controller64can process or operate upon the data received from the linear array60in order to provide compression or encoding, as is known in the art. Generally, the linear array60and the controller64can be of the type commonly found in scanners, fax machines, digital copiers and the like.

Various means can be used to provide successive visual portions of the writing surface14A to the linear array60. As illustrated inFIG. 4, a panning motor66is coupled to a housing68or other structure used to support a lens70and the linear array60. The panning motor64rotates the housing68in order to scan the writing surface14A. For uniformity and simplicity, the angular velocity can be a constant. However, the velocity can also be varied. In another embodiment as illustrated inFIG. 5, the linear array60is displaced linearly within the housing68by a motor69similar to a scanner. In this embodiment, the housing68and the lens70remain substantially fixed. In yet another embodiment as illustrated inFIG. 6, a motor73rotates a mirror74to reflect successive portions of the writing surface14A to the linear array60. These techniques can be used by themselves or in combination to direct the visual sensor13to one or more writing surfaces14A,14B and surface15.

The image data from the sensing device60can be temporarily stored by the controller64and then transferred to the image processor20. Alternatively, the image data can be transferred immediately to the image processor20, where it must be stored as it arrives.

FIG. 7illustrates a method80of installation of the image capturing system10. Step82includes installing the visual sensor13at a fixed location to obtain visual images of the desired writing surface14A. Typically, optical distortion may exist in the visual image sensed by the visual sensor13, depending upon the relative location of the visual sensor13with respect to the writing surface14A. One common form of optical distortion includes “keystoning” where a rectangular writing surface14A may result in a trapezoidal or other non-rectangular visual image. Other forms of optical distortion include lens distortions; distortions generated by the scanning setup inFIG. 4where the distance to particular image portions, and hence their size on the image sensor, vary across the writing surface; ghost images (due to misalignment of R,G and B sensors: can be fixed by detection with convolution followed by subtraction of ghosts); blurring (due to lack of resolution of sensors-can be partially corrected by multiple scans, also called hyper-resolution); and sensor saturation (cannot be fixed, but can be avoided by calibration, or multiple scan at various sensitivity levels).

Step84comprises measuring any optical distortion and providing processing values therefore for each writing surface14A,14B or predefined surface15. The processing values can be stored either in the controller64in a storage device such as memory, or in the image processor20in any of the storage devices described above. Referring toFIG. 8, step84can include placing markers88A,88B,88C and88D or other identifiers on each writing surface14A,14B or predefined surface15in a pattern. For example, the markers88A-88D can be used disposed at the corners of a rectangle. (Note, for distortions other than keystoning, like lens and scanning distortions, four or more markers, for example, nine equally spaced markers may be needed to obtain corrections.) The visual sensor13is then operated to obtain a visual image of the writing surface14A or predefined surface15with the markers88A-88D. In one embodiment, the visual image obtained for the visual sensor13can be displayed upon the monitor47. The user can then identify each of the markers88A-88D as input into a program or other routine to calculate the optical distortion and obtain a processing value-or values, which would correct the visual image to yield a true representation of the writing surface14A or predefined surface15. As used herein, a “processing value” includes any corrective technique, function or quantity, and therefore should not be construed narrowly. In yet another embodiment, the program or routine can automatically identify the markers88A-88D and calculate the necessary correction value to perform the required optical corrections.

At this point, it should be noted that the lens70of the visual sensor13can include a zoom lens in order to magnify the visual image of the writing surface14A or predefined surface15to efficiently use all available resolution provided by the sensing device60, such as the linear array. If the visual sensor13is used to provide visual images of two or more different areas (e.g. writing surfaces14A and14B and/or predefined surface area15), the zoom lens can be motor-actuated wherein a processing value is stored by the controller64indicative of the proper setting for each writing surface14A,14B or area15to be viewed. In another embodiment, a plurality of fixed focal length lenses can be selectively placed (manually or automatically) in the optical path of the visual sensor13in order to provide some optical magnification wherein software routines can be provided to perform additional magnification. Of course, software magnification can also be provided with the zoom lens.

Step90represents an optional step of obtaining reference visual images of the writing surface14A under different lighting conditions. In particular, in view that the writing surface14A,14B or predefined surface area15may have different levels of an incident light from lights within the room or from outside conditions through windows, it may be helpful to obtain one or more reference visual images. For instance, one reference visual image can be obtained on a bright sunny day with the window shades open and with all the lights in the room on. Another reference visual image may be obtained for the same conditions except with the window shades closed. Likewise, reference visual images can be obtained on cloudy days, at night and at various levels of room lighting. Reference visual images can also be obtained at different times during the day if incident light upon the writing surface or other surface to be scanned changes throughout the day.

The reference visual image or images obtained at step90can be used in order to extract information provided on the writing surface14A. For instance, when a visual image is obtained of the writing surface14A having information present thereon, one of the reference visual images can be compared with the current image in order to extract the information. In many cases, the writing surface14A only contains a limited amount of text, numbers, shapes or other information whereas a majority of the writing surface14A is not used and therefore remains its unchanged or background color. By comparing the current visual image with a reference visual image of the writing surface14A where no information is present, the information (i.e. the foreground) of the current visual image can be easily obtained. In one exemplary embodiment for a whiteboard having a white surface, information written on the writing surface will correspond to less light intensity sensed by the sensing elements of the linear array60than that of the background. Corresponding values can be associated for different levels of light intensity. A comparison then can be performed by subtracting the value of a sampled portion of the current visual image (e.g. the value of one sensing element of linear array60for one linear portion of the visual image) from the corresponding value of the sampled portion of the reference visual image. If the resulting value exceeds a threshold, information can be considered present for the sampled portion of the current visual image. In this manner, the current visual image is compared to the reference visual image element by element to obtain the information on the writing surface14A. Depending on the lighting conditions or the time of day, the closest reference visual image can be used in order to minimize effects of bright areas, glares, shadows, etc.

At this point it should be noted that compression can be provided for distribution of the images. As appreciated as those in the art, writing surfaces such as whiteboards are mostly white, and the information written thereon is mostly lines of uniform colors. A good compression algorithm for this type of image could be to separate the image into a foreground (the ink) and a background (the white) with a binary image, called a mask, which tells if a pixel belongs to the foreground (1) or the background(0). Algorithms for computing the mask can be as simple as a fixed threshold on the luminance (above the threshold is foreground and below is background) or more complex, such as the algorithms used in DjVu™, for example, as described in “High Quality Document Image Compression with DjVu”, Journal of Electronic Imaging, 1998 and “Browsing Through High Quality Document Images with DjVu”, In Proc. of Advances in Digital Libraries 98. IEEE, 1998, which are hereby incorporated by reference in their entirety. The binary image can be compressed using standard FAX compression. Because the mask is binary and is typically made out of smooth lines, the mask has an ideal distribution for an arithmetic encoder, such as used in FAX compression. The foreground and the background are smooth color images with don't care conditions on the masked part. The binary image can be compressed with “masked wavelets” (i.e. a wavelet representation which agrees with the image on the “do care” pixels, and have values on the “don't care” pixel in order to maximize compression). For instance, the background can be set to white on its “don't care” pixel, resulting in a mostly white image with smoothly varying shades depending on the lighting. Such an image compresses very well with wavelets. Similarly, the foreground has mostly “don't care” pixels, which can be set to the color of the pen in the region surrounding ink, leading to images which are mostly uniform in large regions.

The reference visual images can also be used to calibrate the sensing device60for various lighting conditions. In particular, processing values such as gain adjustments or sampling speeds (time duration of exposure of the sensing element) of the sensing device60can be calculated for each reference visual image and stored for later use with each visual image having information to obtain. The appropriate stored gain adjustments or sampling speeds can then be retrieved based on the surface14A,14B or15being imaged, the current lighting conditions, and/or the time of day, and applied to the current visual image to improve performance.

In the embodiment illustrated inFIGS. 4-6, the sensing device60comprises a linear array of sensing elements that is exposed to portions of the writing surface14A,14B and surface15successively. The gain adjustments or sampling speeds of the sensing elements can be stored as control values in a storage device accessible by the controller64. The controller64can make gain adjustments and sampling speed adjustments to all the sensing elements as a group for each image portion of visual image. Alternatively, gain adjustments and sampling speed adjustments of the sensing elements can be made in smaller groups of the sensing elements in the array for each image portion, or even individually for each image portions.

In another mode of operation, the reference visual image is obtained before each visual image that is used to record information on the writing surfaces14A,14B or the predefined surface area15. In this manner, the gain adjustments and/or sampling speeds can be calculated and adjusted based on the current lighting conditions. If desired, the reference visual image could be taken more quickly (i.e. lower resolution) than a normal visual image. Also, the reference visual image can be considered valid for a selected time period, wherein if a visual image is desired after the time period has elapsed, a new reference visual image can be obtained. As appreciated by those skilled in the art, the reference visual images discussed herein can include some written information on the writing surface and need not be completely devoid of any markings, etc. The gain adjustments and/or sampling speeds can be calculated as the mean or other statistical function of response of the sensing device60to provide the reference visual image.

The gain adjustments and/or sampling speeds can also be calculated based on samples taken from a plurality of locations94A,94B,94C,94D,94E,94F,94G,94H,94I, and94J (herein exemplified as ten locations) on the writing surface14A,14B or predefined surface15such as illustrated in FIG.9. The gain adjustments and/or sampling speeds for other locations on the writing surface14A,14B and predefined surface area15can then be interpolated and/or extrapolated based on the plurality of locations94A-94J. If any of the plurality of locations94A-94J is suspected to include information (foreground material), the location can be omitted. Another possibility is to identify the foreground colors in a color histograms (the foreground colors will appear as sharp peaks). The pixels identified as foreground can then be replaced by the surrounding background color to yield a “cleaned” reference background image. In this manner, previous visual images could be used as a reference visual image for calculation of gain adjustments and/or sampling speeds for a later visual image.

It should also be noted that a visual image of the writing surface14A,14B or predefined surface area15need not be obtained from only one scan or image thereof. In other words, the visual image of the writing surface14A,14B or predefined surface area15can be formed from a composite of a plurality of separate scans or images. As illustrated inFIG. 9, the writing surfaces14A,14B or predefined surface area15may include areas of bright intensity95, for example, from lights or windows, or areas having shadows97. From the reference visual image or the current visual image, these areas can be identified from the response of the sensing device60, for example, when light intensity exceeds or falls below selected thresholds. The gain adjustment and/or sampling speed can then be adjusted to compensate for one of the identified areas, and the writing surface14A,14B or predefined surface area15, or portion thereof, can be rescanned or reimaged and stored. Additional visual images can be obtained with appropriate compensation provided for each identified area. Since each visual image is taken from the same fixed position, and since the writing surfaces14A,14B and predefined surface area15are also fixed, multiple visual images can be easily combined to form a composite with greater clarity than any one scan or image.

The image capturing system10can be initiated manually, at the visual sensor13, at the image processor20, and/or with a remote, possibly, handheld switch.FIG. 10is an exemplary top plan view of a remote handheld switch110. A first button112can be used to indicate to the visual sensor13that a visual image of writing surface14A should be obtained, while a second button114can be used to indicate that the visual sensor13, or a portion thereof such as mirror, should be directed at the second writing surface14B or predefined surface area15to obtain a corresponding visual image. Of course, additional buttons can be provided for other imaging areas.

If desired, the visual sensor13can be operated so as to obtain periodic visual images of the writing surface14A. An indicator116(FIG. 1) such as a light or a speaker on the visual sensor13, the image processor20or the switch110can be used to indicate to users in the room11that the visual sensor13is going to obtain a visual image shortly, or that the visual sensor13is obtaining a visual image. If periodic visual images are obtained, a software routine can be provided to compare the current visual image with one or more prior images and keep only those that have changed. For instance, the subtraction technique described above can be used wherein suitable thresholds are provided to detect new information including erasures.

In summary, the image capturing system10enables information to be easily obtained from a writing surface or predefined surface area. The information can be stored in any suitable format for transfer to other computers or applications. If desired, optical character recognition can be performed on the textual portions of the information. Likewise, other portions of the information having simple drawings can be converted into known drawing formats.