Patent Publication Number: US-7721204-B2

Title: Client dependent image processing for browser-based image document viewer for handheld client devices

Description:
BACKGROUND OF THE INVENTION 
   1. Field of Invention 
   This invention relates to image processing as a function of the image viewing client device. 
   2. Description of Related Art 
   A user typically likes to view a document on a handheld wireless device before deciding to print, facsimile or e-mail the document. A handheld wireless device, such as, for example, a personal digital assistant (PDA), a cellular phone or a miniaturized computing device, has relatively little screen surface area for viewing documents, as well as little memory for storing the document or the applications for all the document formats a user can conceivably access and view. 
   Current solutions either restrict the document types that can be viewed on handheld client devices, e.g., web pages with graphics interchange format (GIEF) files, show only the ASCII text of the document, provide viewers for a small set of document types, or do not provide viewers at all. In these current solutions, the documents can exist on the handheld client device, a server, e.g., mail server, or any document file/management system in their environment. Generally, documents that do not reside on the handheld client device are typically too large to quickly and/or efficiently transmit over today&#39;s wireless networks. 
   SUMMARY OF THE INVENTION 
   A large portion of the information present in typical documents may be “best” retrieved by using the document image, i.e.: the visual representation, rather than the ASCII representation of the information. This approach is obvious for scanned data, where no ASCII representation is available unless OCR has been additionally performed. However, using the document image to retrieve information in the document may also extend to spreadsheets, where the spatial organization is important; letterheads for telephone numbers, dates, and other contact data; and PowerPoint presentation slides, where almost all information is spatially organized; and the like. Any graphic, table, picture or drawing has an important spatial organization, which can be seen from the two pages  11  and  12  of an example patent document  10  in  FIG. 1 . 
   Generally, the document has to be represented on a small screen of the handheld client device, such as, for example, 96×40, 160×160 or 240×320 pixels. Here, the exact number of pixels is of lower importance, what is important that the physical and pixel-wise dimensions of the display are smaller than the dimensions commonly used on workstations and the like. One approach is to scale the document page image to the client device screen, and, on zooming, scale the appropriate document image regions to the screen. When displaying images on a small screen of a handheld wireless device, an obvious trade-off has to be made between image quality and transmission speed. 
   This invention provides methods and systems that replace the image scaling approach above with client dependent image processing that is a function of at least one or more of a client device parameter, data transmission capability, document type and/or format, and user input. 
   This invention provides methods and systems that create image views of documents that can be used for document identification and data retrieval on handheld client devices. 
   In various exemplary embodiments, this invention provides methods and systems that actively modify processing parameters of an image as a function of the image viewing client. 
   In various exemplary embodiments, this invention provides methods and systems that dynamically control and/or direct document image processing based on client device display capabilities, image formats supported by the client device, network bandwidth, document file format, document content, user hints and the like. 
   These and other features and advantages of this invention are described in, or are apparent from, the following detailed description of various exemplary embodiments of the systems and methods according to this invention. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
     Various exemplary embodiments of the systems and methods of this invention will be described in detail, with reference to the following figures, wherein: 
       FIG. 1  illustrates an example patent document image showing the spatial organization of various text and graphics portions of the document; 
       FIG. 2  illustrates an exemplary embodiment of a system topology that the systems and methods of this invention are usable with; 
       FIG. 3  is a flowchart outlining one exemplary embodiment of a method for processing data according to this invention; 
       FIG. 4  is a flowchart outlining in greater detail one exemplary embodiment of various client dependent image processing operations performed by the server according to the method of  FIG. 3 ; 
       FIG. 5  illustrates an example of the type of image quality that may be obtained by various exemplary embodiments of the client-based image processing methods and systems according to this invention; and 
       FIG. 6  is a functional block diagram of an exemplary embodiment of a client-based image processing system according to this invention. 
   

   DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
   Various exemplary embodiments of systems and methods according to this invention relate to client dependent image processing for browser-based image document viewer for handheld client devices. 
     FIG. 2  shows one exemplary embodiment of a system topology  100  associated with image processing according to this invention. As shown in  FIG. 2 , the system  100  includes one or more servers  110  connected to one or more clients  120  via one or more networks  130 . 
   The server  110  processes image data of a document based on one or more of client-based parameters, transmission network capability, document type and user input requests. The server  110  then compresses the processed data, and transmits the compressed data to the client  120  by way of the network  130 . 
   In various exemplary embodiments, the server  110  receives requests from the client  120  for processing image data and transmitting processed image data back to the client  120 . In various exemplary embodiments, the server  110  processes the image data based on a selection of parameters prestored at the server  110  regarding the capabilities or requirements of the client  120  and the network  130 . In various other exemplary embodiments, the server  110  processes the image data based on the selection of parameters detected from the client  120 , the network  130 , or a combination thereof. 
   In various exemplary embodiments, the server  110  processes the image data based on client device/handset display capabilities, image formats supported by the client device/handset, network bandwidth, document file format, document content, user hints and the like. In various other exemplary embodiments, the server  110  receives the selection of parameters sent from the client  120 , based on various user inputs. 
   As part of the client-dependent image processing operations, the server  110  also compresses the processed image data before transmitting the image data to client  120 . In various exemplary embodiments, the server  110  compresses the processed image data using custom color maps, as described in detail in U.S. patent application Ser. No. 10/901,333, now U.S. Pat. No. 7,539,341, entitled “System and Methods for Processing Image Data Prior to Compression”, which is incorporated herein by reference in its entirety. 
   The server  110  is generally associated with a computer, as described in greater detail below in connection with  FIG. 6 . The client  120  is a device that can display a form of the processed and compressed image data. The client  120  receives the processed and compressed image data from the server  110  via the network  130 . In various exemplary embodiments, the client  120  sends a request for the image data to the server  110  before receiving the processed and compressed data. In various exemplary embodiments, the client  120  selects and sends one or more parameters to the server  110  as a condition for the server  110  to process the data before processing the data. 
   In various exemplary embodiments, the client  120  is a mobile device, such as a laptop computer, a personal data assistant (PDA), a cellular phone, an electronic computing graphics tablet-like device or other handheld computing device and the like. 
   The network  130  transmits data, requests or other information between the server  110  and the client  120 . In various exemplary embodiments, the network  130  is a wired network. In various other exemplary embodiments, the network  130  is a wireless network. In various other exemplary embodiments, the network  130  includes both wired and wireless networks. In various exemplary embodiments, the network  130  includes the Internet. A user may view processed data on a web page displayed on the client  120 . In various other exemplary embodiments, the network  130  includes a wireless cellular phone network or any other existing or yet to be invented communication network. A user may view the processed data displayed on a cellular phone at the client  120 . 
   It will be noted that the following discussion does not discuss in detail all the client-dependent image processing operations that may be performed, or that are possible, according to various exemplary embodiments of systems and methods of this invention. The image processing operations/effects that are discussed below are well known in the art and are relatively straight forward. For example, if the client only supports GIF image formats, only those files will be created. Other image processing operations/effects, such as, for example, crop windows, image orientation, and the like, may also be practiced according to various exemplary embodiments of the systems and methods of this invention. 
     FIG. 3  is a flowchart outlining one exemplary embodiment of a method for processing data according to this invention. As shown in  FIG. 3 , the image processing components that are influenced by client data and that will be described in subsequent sections. It is obvious that the client dependent image processing will lead to a different image processing of the same page/document if viewed by two different client devices (clients) having different capabilities or attributes. It should also be noted that two different documents will undergo different image processing even for the identical client based on document attributes. It is this second aspect that the various exemplary embodiments of the methods and systems according to this invention will discuss in detail below. 
   As shown in  FIG. 3 , beginning in step S 110 , operation of the method continues to step S 120 , where a server receives a request for a document from a client via a network. In various exemplary embodiments, the request is a signal generated when a user clicks an icon/link displayed at the client. The icon/link represents document data to be processed. 
   Next, in step S 130 , the server processes the document data. The document data may be processed based on one or more of client device/handset display capabilities, image formats supported by the client device/handset, network bandwidth, document file format, document content, user hints and the like. 
   Following the processing of the document data, at step S 140 , the client-dependent processed data is transmitted to the client device for display on the client device. Then, operation of the method continues to step S 150  where the operation stops. It should be noted that all standard methods of anticipating user requests can be performed on the server, based on configuration, load, etc., thereby reducing the latency for subsequent requests without altering the spirit of the invention. 
     FIG. 4  is a flowchart outlining in greater detail one exemplary embodiment of various client dependent image processing operations performed at step S 130  by the server according to the method of  FIG. 3 . In various exemplary embodiments, the server based image processing operations include the steps of running the document application in order to create a data file that can be ‘RiPped’ into an image raster. All raster operations are then performed, such as, for example, rotation, scaling, cropping, enhancement, and the document image data is transformed into a client native format. For standard browser applications, this is either GIF, JPG or WBMP (WAP) format in the suitable size. In various exemplary embodiments, the server based image processing operations include the steps described in detail in U.S. patent application Ser. No. 10/901,241, now U.S. Pat. No. 7,620,892, entitled “Server Based Image Processing For Client Display of Documents”, which is incorporated herein by references in its entirety. 
   As shown in  FIG. 4 , the client dependent image processing operations begin at step S 130  and continue to step S 1310  where the requested document is converted to a raster image on the server using well known raster image processing (RIP) techniques. This raster image generally uses some client capabilities, such as, for example, black and white rendering for a black and white device, and some document attributes, for example, lower initial resolution for PowerPoint documents compared to Word documents. The use of document attributes can be understood when considering the likelihood that certain document types will have certain properties. Presentation slides, for example, will have larger fonts and thus generally require a lower initial RIP resolution than legal documents. 
   Next, at step S 1320 , the ‘RIPed’ data is subsequently scaled to the desired zoom settings. For example, in various exemplary embodiments, the data may be scaled to one of the current prototype mDoc version 4 zoom settings, e.g., 2, 4, 8 and 16 reduction, using a density preserving reduction (filter &amp; down sampling) leading to 5 raw images at the 5 different resolutions. The raw images may not yet adhere to the client color and bit-depth requirements. 
   Following data scaling, operation of the method continues to step S 1330  where client-dependent image filtering is performed. At high reduction levels, the page content can only be “guessed” by the user. Twelve point text, for example, would be rendered at single pixel height for most letters. In various exemplary embodiments of the methods and systems according to this invention, the image sharpness is strongly increased for higher reduction factors, well above any limit that would be used for normal image viewing. However, increasing the sharpness at high reduction factors gives the user the perception of “text” without making the text readable. For the exemplary prototype mDoc application, speed is an important consideration as it directly influences user latency. As a speed/quality compromise, in one exemplary embodiment, a 5 element filter was chosen where four ring values are constant (“−1”) and the center value is varied. Smaller center values yield stronger enhancement. Other techniques known in the art or later developed may also be used to perform client dependent filtering operations. 
   After image sharpening, at step S 1340 , enhancement TRCs are used to further increase the contrast of the data. The TRCs are a function of reduction factors (stronger enhancement for higher reductions) and of document applications. In this case, presentation slides and scanned data undergoes an essential symmetric contrast enhancement. Text documents and spreadsheets undergo a non-symmetric enhancement leading to a strong visual darkening. This is based on the likelihood that different color and foreground/background schemes are used in the different document types. 
   At step S 1350 , a data compression operation is performed. As part of the data compression operation, data compression parameters may be varied based on transmission bandwidth and client storage capabilities, as well as other factors. In the binary case, no variation can be obtained at this step directly, but rather a different enhancement can be performed. For color clients, a choice exists between JPEG and GIF compressions, since both are normally supported in the browser. In JPEG compression, the Q-table/Q-Factor can be used to change compression ratios. For the “GIF compression” (better in the GIF format encapsulated compression method), the compression efficiency may be modified by the initial color mapping. For the discussion presented below, a dynamic color mapping between “222” (8 colors) and “794” (252 colors) is assumed, as described in detail in U.S. patent application Ser. No. 10/901,333, entitled “Systems and Methods For Processing Image Data Prior To Compression”, which is incorporated herein by reference in its entirety. 
   Following data compression, the method continues to step S 1360  where a client dependent data formatting operation is performed. Following formatting of the data, the client dependent processed document/image is transmitted to the client device, step S 140 . 
   Table 1 is an exemplary embodiment of an image processing table showing the different image processing aspects as a function of reduction (zoom setting) and application. In Table 1, TRC name 64 L indicates a TRC that is non-symmetric (“L”eft) and that maps all pixel values below 64 to “black”. A TRC name 32 indicates a symmetric TRC that maps all values below 32 to black and all values above 223 to white. The rest of the range is a linear mapping between black and white. The term “reduce” refers to a filtered subsampling, preferably maintaining image density attributes. 
   
     
       
         
             
             
             
             
             
           
             
                 
               TABLE 1 
             
             
                 
                 
             
             
                 
               Word 
                 
                 
                 
             
             
                 
               processor 
               Spreadsheet 
               Presentation 
               Scan 
             
             
                 
                 
             
           
          
             
                 
             
          
         
         
             
             
             
             
             
          
             
               RIP 
               150 
               150 
               75 
               — 
             
             
               Resolution 
             
             
               Scale Method 
               Reduce 
               Reduce 
               Reduce 
               Reduce 
             
             
               Zoom level 
               1, 2, 3, 4 
               1, 2, 3, 4 
               1, 2, 3 
               1, 2, 3, 4 
             
             
               Enhance 
               TRC 
               TRC 
               TRC 
               TRC 
             
             
               Method 
               stretching 
               stretching 
               stretching 
               stretching 
             
          
         
         
             
             
             
             
             
             
             
             
             
          
             
               Enhance TRC 
               1 
               NO 
               1 
               NO 
               1 
               NO 
               1 
               32 
             
             
               For different 
               2 
               32L 
               2 
               32L 
               2 
               32 
               2 
               50 
             
             
               scale factors 
               3 
               64L 
               3 
               64L 
               3 
               50 
               3 
               64 
             
             
               Linear 
               4 
               100L  
               4 
               100L  
                 
                 
               4 
               64 
             
             
               between: 
             
          
         
         
             
             
             
             
             
          
             
               Filter method 
               5 Element 
               5 Element 
               5 Element 
               5 Element 
             
             
                 
               sharpen 
               sharpen 
               sharpen 
               sharpen 
             
          
         
         
             
             
             
             
             
             
             
             
             
          
             
               Center Value 
               1 
               NO 
               1 
               NO 
               1 
               NO 
               1 
               NO 
             
             
                 
               2 
               8 
               2 
               8 
               2 
               8 
               2 
               8 
             
             
                 
               3 
               8 
               3 
               8 
               3 
               8 
               3 
               8 
             
             
                 
               4 
               6 
               4 
               6 
                 
                 
               4 
               6 
             
             
                 
             
          
         
       
     
   
   As can be observed from Table 1, different image processing settings are used for different document types and zoom levels. The exemplary values given in Table 1 are currently used in an exemplary mDoc viewer demo produced by the inventors. For binary clients, such as for example, WAP phoned, processing stops at the level of Table 1, with only the file format conversion to WBMP being done after the image processing. For color capable clients, some additional steps, which are well known in the art, may be added. 
     FIG. 5  illustrates an example of the image quality that may be obtained using the various exemplary embodiments of the client-based image processing methods and systems according to this invention. The image on the left of  FIG. 5 , e.g., reference image  55 , shows as reference the standard filtered subsampling. The reference image was done using a pre-filtered subsampling using the identical reduction algorithm as the one used in the exemplary methods and systems according to this invention. 
   The reference image shown on the right of  FIG. 5 , e.g., reference image  56 , has been client-dependent processed using the various exemplary methods according to this invention. It will be noted that the image has not been cropped to the screen size. That is,  FIG. 5  shows a different zoom level without cropping to the client window. 
   The client dependent image processing controls discussed above show a clear image quality improvement compared to standard, fixed processing. The processing overhead on a standard personal computer is small compared to the standard transmission times over the wireless network, and generally can be ignored. One possible exception to this may be the speed of the sharpness filter, leading to its omission for the largest images (full zoom). 
     FIG. 6  is a functional block diagram of an exemplary embodiment of a client dependent image processing system according to this invention. As shown in  FIG. 6 , the client dependent image processing system  800  typically resides in a server environment. 
   As shown in  FIG. 6 , the data processing system  800  may include an input/output (I/O) interface  820 , a controller  830 , a memory  840 , a parameter selecting circuit, routine or application  845 , an application library  847 , a raster image processing circuit, routine or application  850 , a document data scaling circuit, routine or application  860 , a client dependent image filtering circuit, routine or application  870 , a client-dependent image enhancement circuit, routine or application  880 , a data compression circuit, routine or application  890 , and a client dependent data formatting circuit, routine or application  895 , each interconnected by one or more control and/or data buses and/or application programming interfaces  900 . 
   In various exemplary embodiments, the data processing system  800  is implemented on a programmable general purpose computer. However, the data processing system  800  can also be implemented on a special purpose computer, a programmed microprocessor or microcontroller and peripheral integrated circuit elements, an ASIC or other integrated circuits, a digital signal processor (DSP), a hard wired electronic or logic circuit, such as a discrete element circuit, a programmable logic device such as a PLD, PLA, FPGA or PAL, or the like. In general, any device capable of implementing a finite state machine that is in turn capable of implementing the flowchart shown in  FIGS. 3 and 4  can be used to implement the data processing system  800 . 
   The input/output interface  820  interacts with the outside of the data processing system  800 . In various exemplary embodiments, the input/output interface  820  may receive input from the outside, such as a network, via one or more links  810 . The input/output interface  820  may output data for subsequent delivery or transmission via the one or more links  810  to a client device (shown in  FIG. 1 ). 
   The one or more links  810  may be direct connections, or may be implemented using a connection device such as a modem, a local area network, a wide area network, an intranet, the Internet, and any other distributed processing network, or any other known or later-developed connection device. 
   As shown in  FIG. 6 , the memory  840  includes a parameter default selection portion  841  and one or more client dependent image processing table(s)  842 . The parameter default selection portion  841  stores default selections of parameters for client devices (“clients”). The parameter default selection portion  841  can also store parameters of various networks and display devices. The one or more client dependent image processing table(s)  842  stores different image processing settings that are used for different document types, zoom levels, and the like. 
   The memory  840  may also store any data and/or program necessary for implementing the functions of the data processing system  800 . The memory  840  can be implemented using any appropriate combination of alterable, volatile, or non-volatile memory or non-alterable or fixed memory. The alterable memory, whether volatile or non-volatile, can be implemented using any one or more of static or dynamic RAM, a floppy disk and a disk drive, a writable or rewritable optical disk and disk drive, a hard drive, flash memory or the like. Similarly, the non-alterable or fixed memory can be implemented using any one or more of ROM, PROM, EPROM, EEPROM, an optical ROM disk, such as a CD-ROM or a DVD-ROM disk and disk drive or the like. 
   In the exemplary embodiments of the data processing system  800  shown in  FIG. 6 , the parameter selecting circuit, routine or application  845 , under the control of controller  830 , determines a selection of parameters. In various exemplary embodiments, the parameter selecting circuit, routine or application  845  determines the selection of parameters based on the specific characteristics of the client device, a user&#39;s instructions, a default selection stored in the memory  840 , and/or detected features of the network, the client and/or the document type. In various exemplary embodiments, a detection may be made by detecting a network and/or device type, coupled with parameters stored for that network/device type. In various exemplary embodiments, different documents are processed differently. For example, a Word™ document is processed differently from a PowerPoint™ document, with the corresponding application files being rendered using appropriate application software from library  847 . In various exemplary embodiments, different client devices process the same document differently. 
   The raster image processing circuit, routine or application  850 , under the control of controller  830 , converts a document requested by the client to a raster image on the server using well known raster image processing (RIP) techniques. 
   The document data scaling circuit, routine or application  860 , under the control of controller  830 , scales the ‘RIPed’ data to the desired zoom settings. For example, in various exemplary embodiments, the data may be scaled to one of the current prototype mDoc version 4 zoom settings, e.g., 2, 4, 8 and 16 reduction, using a density preserving reduction (filter &amp; down sampling) leading to 5 raw images at the 5 different resolutions. 
   The client dependent image filtering circuit, routine or application  870 , under the control of controller  830 , performs client-dependent image filtering of the scaled data. In various exemplary embodiments of the methods and systems according to this invention, the image sharpness is strongly increased for higher reduction factors, well above any limit that would be used for normal image viewing. 
   The client-dependent image enhancement circuit, routine or application  880 , under the control of controller  830 , uses enhancement TRCs to further increase the contrast of the data. 
   Following image enhancement, the data compression circuit, routine or application  890 , under the control of the controller  830 , compresses the processed data. In various exemplary embodiments, the data compression circuit, routine or application  890  compresses the processed image data using tonal reproduction curves (TRC) to suppress backgrounds and to further enhance the perception of detail and optionally and/or custom color maps, as described in detail in U.S. patent application Ser. No. 10/901,333, now U.S. Pat. No. 7,539,341 entitled “System and Methods For Processing Image Data Prior to Compression”, which is incorporated herein by reference in its entirety. 
   The client dependent data formatting circuit, routine or application  895 , under the control of controller  830 , finally formats the data prior to transmitting the data over the network to the client device. In various exemplary embodiments, the client dependent data formatting circuit, routine or application  895  formats the data based on one or more client dependent image processing table(s)  842 . As discussed above, tables  842  store different image processing settings that are used for different document types, zoom levels, and the like. 
   While particular embodiments have been described, alternatives, modifications, variations and improvements may be implemented within the spirit and scope of the invention.