Recent applications of IBM's query by image content (QBIC)

The QBIC project at IBM's Almaden Research Center is studying methods to query large on-line image databases using the images' content as the basis of the queries. Examples of the content include color, color layout, texture, shape, size, orientation, and position of image objects and regions. We have developed an AIX prototype, WWW QBIC server with WWW demo, and a product called Ultimedia Manager (OS/2 and Windows). The key applications of this technology are in the areas where image patterns are the basis of the queries, and where keywords are not precise enough for satisfactory retrieval, such as in on-line stock photo, retail, art etc. In this paper we briefly outline several recent applications of QBIC. SEDONA Systems Corporation, PIP Printing and IBM announced an application where QBIC is used to help the user find images to be included in documents remotely printed in high resolution and quality; IBM in the UK is developing an "advisor" to textile designers; Prof. B. Holt (UC Davis) researches use of QBIC for retrieving art images based on examples of a motifs used across geography and time without having to use standard authority lists of text reference terms, especially when searching for visual elements that are difficult to describe in words; and Prof. J. Hethorn (UC Davis) uses Ultimedia Manager to study colors, styles and trends in fashion. In this paper we "Permission to make digital/hard copy of all or part of this material without fee is granted provided that copies are not made or distributed for profit or commercial advantage, the ACM copyright/server notice, the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery, Ine. (ACM). To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. " present a short overview of QBIC technology and products, and summarize the above applications. Introduction to Query by Image Content (QBIC) Today's technology enables users to acquire, store, manipulate, and transmit large numbers of images. However, finding the right image in a large image database is a difficult problem. Traditional database retrieval technology uses an "exact match" approach, which works well for traditional structured information. However, for multimedia data types, such as digitized images, this approach breaks down. More and more we are witnessing applications where very large image collections are available and where exact search is becoming increasingly inadequate. Some examples of these applications are in retail cataloging, stock photography and large collections of art and graphics. The current approach is to index and then search the images based on some identifying text such as title, descriptive keywords (from a limited vocabulary), or catalog numbers. Some programs offer full relational database (SQL) search over relational tables, where each column can contain text or numeric data. These systems are useful in many applications where exact keywords and numeric descriptors can be easily obtained and are sufficient to find images the user wants. However, more powerful searches will be necessary. We have developed a way to index and search images by the content of the image. Database queries can select for color, texture, shape and position (1,2,3,4). This approach is called QBIC, for Query by Image Content. Ultimedia Manager is an OS/2 and Windows product from IBM that combines QBIC technology with traditional database searching, and offers a powerful image browser. This unique combination provides new ways to manage image databases. The QBIC portion of Ultimedia Manager can index and retrieve images according to the following kinds of information: Average color; Histogram color (based on 64 predefined colors); Texture or Pattern; Shape (of user outline); and Position. These measures can be calculated for the whole image or for user-defined areas within the image. These areas are considered to be "visual indexes" to the image. Part of the classification (indexing) process is for the user to identify areas of interest within the image. This, for example, allows a user to search for a red flower in an image of green foliage. When running a query, Ultimedia Manager ranks each image based on a distance measurement computed between the query attribute values and the corresponding features or attributes stored in the QBIC Catalog database. This is done only for images that were selected by exact data search (that is, SQL). Users can modify the relative weights of each visual attribute to satisfy their preferences in influencing the final sorting of result images. See Fig. 1 that shows SQL data selector, color histogram selector, query window and result area window of Ultimedia Manager, in an application of searching through fashion catalogs. QBIC has an increasing importance in the new world of WWW. There is a strong interest among content providers and retailers to provide their collections (for sale or for browsing) over the WWW. Now, imagine having a large on-line collection of photographs and posing a keyword/parametric query such as "give me all SHIRTS made of COTTON in the PRICE range 10 to 20 dollars". This query may produce large number of responses, which makes it impractical for browsing since thumbnails are obtained through standard WWW connections (modems). (After all, we do not buy shirts by only specifying price and material either). Thus, one of the basic functions of image searching, namely fast and easy to use browsing, can not be performed on the WWW due to low ly~ndwidth. Now, imagine that in applications where image appearance is important (on-line stock photo, retail, fashion, art) one can specify some kind of QBIC query, like a set of colors. QBIC, running at the server, can sort the images that satisfy the initial keyword/parametric query and thert send to the browser only a few top matches (in this case having the desired set of colors), thus dramatically reducing the bandwidth requirements. A user can either be satisfied with this or use relevance feedback to change colors or by clicking at the image to ask for "more images like this". This way an easy to use and fast search/browse is possible even in the case of slow WWW access. We have developed a QBIC server for WWW with associated APIs and simple GUI, that has been operational since mid 1995 in the form of a demo with 1900 images. It offers color histogram query (selection of desired colors and their amounts) and a new very powerful color layout query which is ideally suited for on-line stock photography (setection of colors and their position in the image by using simple painting program). The current interface uses standard HTML and plans are under way to make it more interactive. This server, part of several IBM customer solutions, is part of the IBM Digital Library where it is integrated with text and parametric search, and is also available as a special deal. Fig. 2 shows our WWW QBIC browser. QBIC applications in stock-photography for remote printing As a result of an agreement between SEDONA Systems Corporation, PIP Printing, and IBM. desktop publishers, graphic designers, and consumers can now quickly and inexpensively search for and print thousands of high-quality, royalty-free images via the lnternet, using QBIC technology. The agreement will enable Internet users to search for stock images and clip art by selecting attributes including color, texture, and shape using IBM°s award-winning Query by Image Content (QBIC) technology. These images can be quickly transmitted to loeat or distant PIP Printing centers using REPRINT, SEDONA's document communications software. The companies announced the new service at the Seybold '95 Conference. Stock images and clip art may be retrieved from the Internet without royalty payments, then resized, rotated, and embedded in