Patent Description:
A trademark is typically a name, word, phrase, logo, symbol, design, image, or a combination of these elements to identify the products or services of a particular source from those of others. Thus, trademarks may include very different types of lines and colors to form words, shapes, patterns, and/or logos. Trademarks are applied for and registered around the world in different countries' government agencies that examine and approve of trademark applications. Often to apply and to be approved for a trademark, a search is conducted to check if similar registered trademarks exist.

Content based image retrieval, or visual search, is the task of retrieving digital images from a plurality of images that are similar with respect to the visual characteristics of some query image. Visual search technology affords several advantages over traditional keyword search. Importantly, it allows users to search for images in collections that have not been tagged with descriptive metadata and to search with an image rather than text, which may be a much richer query than a sequence of keywords.

However, in order to use images for search, image processing is performed to extract, identify, or otherwise recognize attributes or features of the images. <CIT> discloses an apparatus and method for obtaining image feature data of an image. Color features are obtained by processing color data in HSV color space and shape features are obtained by processing each pixel. However, using original highly segmented RGB values and processing entire image to obtain shape features consumes both time and space in processing. relates to content based image retrieval and discloses to detect <NUM> types of patterns based on the orientation of pixels in a digital image. But, focusing on the detection of <NUM> basic patterns in an image is insufficient for adequality comparing trademark images which mainly comprise abstract logos. At present, accurate and efficient search methods and systems have not been available that return quality matching trademarks with sufficient speed. Thus, there is a need for an automatic, computationally-efficient, and accurate method and system for searching, matching, and retrieving images from a set of trademark images.

The present invention addresses these problems by providing a highly efficient and accurate trademark image search and retrieval method, system, and computer product.

Advantageous embodiments are defined by the dependent claims.

Methods, systems, and computer products for trademark searching and retrieval according to the invention and some particular embodiments thereof will be described with reference to the following figures. These and other features, aspects, and advantages of the present invention will become better understood when the following detailed description is read with reference to the accompanying drawings in which like characters represent like parts throughout the drawings. Unless noted, the drawings may not be drawn to scale.

According to the general disclosure, methods, systems, and computer products for searching and retrieving "matched" trademark images based upon the content of a query image and a set of registered trademark images are disclosed. In accordance, a method for image similarity comparison and retrieval of companies' registered trademarks includes at least two stages: (<NUM>) calculation of image color feature similarity; and (<NUM>) calculation of image shape feature similarity.

Described in further detail herein is a method, system, and computer product for extracting image feature data from an input trademark image and various uses of the image feature data. Image feature data includes color histogram, and orientation histogram. The query or input image may comprise a digitized photograph taken by a user to capture an image, such as at least a color and/or pattern, or an image included in a website or web page. The extracted image feature data is used to provide similar trademark images in a provided database that match the query image. In some embodiments, one or more sources are used to obtain sets of trademark images for comparison (e.g., registered trademarks from different national government agencies).

<FIG> illustrates a network diagram depicting an example system <NUM> for performing image processing and using image feature data obtained from image processing for trademark image searching according to some embodiments. A networked system <NUM> forms a network-based publication system that provides server-side functionality, via a network <NUM> (e.g., the Internet or Wide Area Network (WAN)), to one or more clients and devices. <FIG> further illustrates, for example, one or both of a web client <NUM> (e.g., a web browser) and a programmatic client <NUM> executing on device machine <NUM>. In one embodiment, the system <NUM> comprises a matching system, a recommendation system, and or a registration service system.

Device machine <NUM> comprises a computing device that includes at least a display and communication capabilities with the network <NUM> to access the networked system <NUM>. The device machine <NUM> comprises, but is not limited to, remote devices, work stations, computers, general purpose computers, Internet appliances, hand-held devices, wireless devices, portable devices, wearable computers, cellular or mobile phones, portable digital assistants (PDAs), smart phones, tablets, ultrabooks, netbooks, laptops, desktops, multi-processor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, network PCs, mini-computers, and the like. Device machine <NUM> may connect with the network <NUM> via a wired or wireless connection. For example, one or more portions of network <NUM> may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, a wireless network, a WiFi network, a WiMax network, another type of network, or a combination of two or more such networks.

Device machine <NUM> includes one or more applications (also referred to as "apps") such as, but not limited to, a web browser, messaging application, electronic mail (email) application, an e-commerce site application (also referred to as a marketplace application), a trademark search and/or registration application, and the like. In some embodiments, if the trademark application is included in a given device machine <NUM>, then this application is configured to locally provide the user interface and at least some of the functionalities with the application configured to communicate with the networked system <NUM>, on an as needed basis, for data and/or processing capabilities not locally available (such as access to a database of trademark images, to authenticate a user, to verify a method of payment, etc.). Conversely if the trademark search application is not included in a given device machine <NUM>, the device machine <NUM> may use its web browser to access a trademark service site (or a variant thereof) hosted on the networked system <NUM>. Although a single device machine <NUM> is shown in <FIG>, more device machines can be included in the system <NUM>.

An Application Program Interface (API) server <NUM> and a web server <NUM> are coupled to, and provide programmatic and web interfaces respectively to, one or more application servers <NUM>. The application servers <NUM> host one or more "trademark applications" (e.g., trademark service applications <NUM> and trademark search applications <NUM>) in accordance with an embodiment of the present invention. Application servers <NUM> may further include payment applications and other applications that support a trademark service. The application servers <NUM> are, in turn, shown to be coupled to one or more databases servers <NUM> that facilitate access to one or more databases <NUM>.

The trademark service applications <NUM> may provide a number of trademark registration functions and services to users that access networked system <NUM>. Trademark registration functions/services may include a number of trademark registration functions and services (e.g., provision of forms, laws, and/or information; data intake; image capture; payment, etc.). For example, the trademark service applications <NUM> may provide a number of services and functions to users for providing their trademark(s) (e.g., capturing trademark), registering their trademark(s) with a government agency, and offering services for sale (e.g., consulting, interfacing with the government agency to respond to any objections for the trademark registration, facilitating correspondence, and other services related to trademark registration). Additionally, the trademark service applications <NUM> may track and store data and metadata related to captured trademarks, transactions, and user interactions. In some embodiments, the trademark service applications <NUM> may publish or otherwise provide access to content items stored in application servers <NUM> or databases <NUM> accessible to the application servers <NUM> and/or the database servers <NUM>.

The trademark search applications <NUM> may include a number of trademark search and retrieval functions and services (e.g., searching, reporting, review and feedback, and other services or functions related to trademark searching, etc.). The trademark search applications <NUM> may allow users to search a set or database of registered trademarks for similar images to a provided query or input image, which may be of interest for a potential trademark registration application in one example. The trademark search applications <NUM> extracts color and shape features from the query image and a set of searched images (e.g., registered trademark images), generate histograms corresponding to color and oriented gradients, and then compare the histograms between the query image and set of database images for close "matches". The set of searched images may be from database <NUM> or a third party server <NUM> having access to a third party database <NUM> (e.g., publicly available registered trademark images from various countries' government agencies that register trademarks). Additionally, the trademark search applications <NUM> may track and store data and metadata related to captured trademarks, transactions, and user interactions. In some embodiments, the trademark search applications <NUM> may publish or otherwise provide access to content items stored in application servers <NUM> or databases <NUM> accessible to the application servers <NUM> and/or the database servers <NUM>.

While the trademark applications <NUM> and <NUM> are shown in <FIG> to both form part of the networked system <NUM>, it will be appreciated that, in alternative embodiments, the trademark applications may form part of a trademark application service that is separate and distinct from the networked system <NUM> or separate and distinct from one another. In other embodiments, the trademark service applications <NUM> may be omitted from the system <NUM>. In some embodiments, at least a portion of the trademark applications may be provided on the device machine <NUM>.

Further, while the system <NUM> shown in <FIG> employs a client-server architecture, embodiments of the present disclosure is not limited to such an architecture, and may equally well find application in, for example, a distributed or peer-to-peer architecture system. The various trademark service and search applications <NUM> and <NUM> may also be implemented as standalone software programs, which do not necessarily have networking capabilities.

The web client <NUM> accesses the various trademark applications <NUM> and <NUM> via the web interface supported by the web server <NUM>. Similarly, the programmatic client <NUM> accesses the various services and functions provided by the trademark applications <NUM> and <NUM> via the programmatic interface provided by the API server <NUM>. The programmatic client <NUM> may, for example, be a trademark services application to enable users to capture images and/or manage trademark applications on the networked system <NUM> in an off-line manner, and to perform batch-mode communications between the programmatic client <NUM> and the networked system <NUM>.

<FIG> also illustrates a third party server machine <NUM> executing a third party application <NUM>, which has programmatic access to the networked system <NUM> via the programmatic interface provided by the API server <NUM>. For example, the third party application <NUM> may, utilizing information retrieved from the networked system <NUM>, support one or more features or functions on a website hosted by the third party. The third party website may, for example, provide one or more trademark services that are supported by the relevant applications of the networked system <NUM>. As previously noted, in some embodiments third party server <NUM> may be a server for a national government agency that registers trademarks and allows for access to a database <NUM> of registered trademark images.

Referring now to <FIG>, a block diagram illustrates components provided within the networked system <NUM> according to some embodiments. The networked system <NUM> may be hosted on dedicated or shared server machines (not shown) that are communicatively coupled to enable communications between server machines. The components themselves are communicatively coupled (e.g., via appropriate interfaces) to each other and to various data sources, so as to allow information to be passed between the applications or so as to allow the applications to share and access common data. Furthermore, the components may access one or more databases <NUM> via the database servers <NUM>. It is also possible that components may access one or more third party databases <NUM>.

The networked system <NUM> may provide a number of trademark service mechanisms whereby an applicant may provide or capture a query image (e.g., a potential trademark), and the system <NUM> may then compare the received query image to a database or set of images (e.g., registered trademarks), provide best "matches", display returned or retrieved images to allow viewing of the retrieved images, and apply for a trademark registration. To this end, the networked system <NUM> may comprise at least one trademark image capture engine <NUM>, at least one trademark image feature extraction engine <NUM>, at least one trademark image search engine <NUM>, at least one trademark image retrieval/publication engine <NUM>, at least one navigation engine <NUM>, and at least one trademark services engine <NUM>.

The trademark image capture engine <NUM> allows for receiving a query image from a user to be used as the basis of a search by search engine <NUM>. The image may be taken from a camera or imaging component of a client device (e.g., a laptop, a mobile phone, or tablet) or may be accessed from storage. In one example, capture engine <NUM> digitizes or processes the query image.

The trademark image feature extraction engine <NUM> enables extraction of image features, such as color features extracted as a color histogram, and shape features extracted as an oriented gradients histogram. The image features of color and shape can then be used to search for registered trademarks similar to the query image.

The trademark image search engine <NUM> enables image queries of registered trademarks. In example embodiments, the search engine <NUM> receives a query image from a device of a user and conducts a comparison between the extracted image features (e.g., color and shape) and the features of the images in the set of registered trademarks. The search engine <NUM> may record the query and any subsequent user actions and behaviors (e.g., navigations). The search engine <NUM> may also perform a search based on whether the image includes only words, only colors, or both words and colors, and/or whether the image is in black and white or in color.

The search engine <NUM> also may perform a search based on the country of origin of the registered trademarks, database, or agency to be searched. In addition, a user may access the search engine <NUM> via a mobile device and generate a search query. Using the search query, the search engine <NUM> compares relevant image features to find the best matches, in one example based upon the shortest mathematical distance, such as Bhattacharyya distance.

The trademark image retrieval/publication engine <NUM> returns relevant search results for similar or matched registered trademarks based upon the compared color and shape features. In one example, image retrieval/publication engine <NUM> returns results in numerical order of shortest Bhattacharyya distance. Image retrieval/publication engine <NUM> may also include a category or classification for each returned registered trademark (e.g., Nice classification in which the registered trademark is classified under a good or service associated within a particular category). Additional information associated with the registered trademarks, such as owner, registrant, description, registration date, and the like, are within the scope of the embodiment.

Networked system <NUM> may further include a navigation engine <NUM>, which allows users to navigate through various categories or classifications of the retrieved (returned) registered trademarks. For example, the navigation engine <NUM> allows a user to successively navigate down a category tree comprising a hierarchy of categories (e.g., the category tree structure) until a particular set of listings is reached. Various other navigation applications within the navigation engine <NUM> may be provided to supplement the searching and browsing applications. The navigation engine <NUM> may record the various user actions (e.g., clicks) performed by the user in order to navigate down the category tree.

Additional modules and engines associated with the networked system <NUM> are described below in further detail. It should be appreciated that modules or engines may embody various aspects of the details described below.

<FIG> illustrates a block diagram showing image processing and image data usage functionalities / operations implemented in modules and libraries / data structures / databases according to some embodiments. The modules comprise one or more software components, programs, applications, or other units of code base or instructions configured to be executed by one or more processors included in the application servers <NUM>, and/or device machine <NUM>. The modules include a trademark image capture module <NUM>, a color histogram module <NUM>, an oriented gradients histogram module <NUM>, an indexing module <NUM>, a comparison module <NUM>, a trademark image retrieval module <NUM>, a browser plugin module <NUM>, and a user interest detection module <NUM>. The modules <NUM>-<NUM> can communicate with each of a registered trademark image database <NUM> and trademark image index database <NUM>, in which databases <NUM>, <NUM> may or may not be included in the databases <NUM>. It is also noted that database <NUM> may be associated or linked to third-party database <NUM>. Although the modules <NUM>-<NUM> are shown as distinct modules in <FIG>, it should be understood that the modules <NUM>-<NUM> may be implemented as fewer or more modules than illustrated. It should also be understood that any of the modules <NUM>-<NUM> may communicate with one or more components included in the system <NUM>, such as database servers <NUM>, application servers <NUM>, third party server <NUM>, or device machine <NUM>. Similarly, databases <NUM>, <NUM> are shown as distinct databases in <FIG>. However, it is understood that the content of databases <NUM>, <NUM> may be stored in fewer or more databases than illustrated.

In some embodiments, one or more of the modules <NUM>-<NUM> are downloaded from a service site appropriate for the type of computing device, or multiple modules or applications that are OS dependent may be available for download. For example, if the device machine <NUM> comprises an iOS-type device (e.g., Mac, iPhone, or iPad), then the modules (which can be packaged as part of a trademark services and/or search application) can be downloaded from iTunes. Similarly, if the device machine <NUM> comprises an Android-type device, then the modules can be downloaded from the Android Marketplace. The device machine <NUM> has communication capabilities with servers or databases at a remote location (e.g., databases <NUM>, database servers <NUM>, API server <NUM>, web server <NUM>) to access data and/or processing capabilities to facilitate image capture, image processing, and use of image data from image processing as described in further detail below.

In other embodiments, one or more of the modules <NUM>-<NUM> may be hosted on the application servers <NUM> and no download of the modules is required on the device machines <NUM>. Instead, the modules may be accessed by device machines <NUM> using a web browser over the network <NUM>. In still other embodiments, some of the modules may be included in the device machines <NUM> while other of the modules may be included in the application servers <NUM>; the device machines <NUM> communicating with the application servers <NUM> to together provide the appropriate functionalities.

<FIG> illustrates an example flow diagram <NUM> for image processing and determination of image matches implemented by the modules of <FIG> according to some embodiments. The operations of the flow diagram <NUM> may be performed by the device machine <NUM>, and/or a server included in the networked system <NUM> (e.g., API server <NUM>, web server <NUM>, application servers <NUM>, database servers <NUM>).

Operations/functionalities of flow diagram <NUM> can be classified into two phases: an indexing phase 401A and a matching phase 401B. In some embodiments, the indexing phase 401A comprises offline image processing of pre-existing registered trademark images (e.g., images corresponding to registered trademarks stored in database <NUM>, <NUM>, and/or <NUM>) by, for example, application servers <NUM> in order to obtain image feature data. The pre-existing registered trademarks may be provided from various sources, such as different government agencies, a privately compiled database of registered trademarks, or the like.

The obtained image feature data can be indexed into the image index database <NUM> and subsequently used for faster look-up during the matching phase. In some embodiments, the registered trademark images can be processed in one or more batches. In some embodiments, a part or a batch of the registered trademark images can be processed online and the obtained image feature data can be indexed into the image index database <NUM>. Images may be indexed based on their features/attributes such as, but not limited to, color distribution shown through a color histogram, orientation histogram, and the like, and also based on other available information associated with the registered trademark, such as but not limited to, Nice classification, registrant, owner, country, and the like.

Once the indexing phase 401A is complete, the matching phase 401B can be triggered by receipt of a query comprising an image (referred to as a query image or an input image). The image index database <NUM> is accessed to find the closest matching registered trademark image(s). These registered trademark image(s) are presented as "matching" results to the query image based upon a similarity measure. Blocks <NUM> - <NUM> relate to the indexing phase 401A, and blocks <NUM> - <NUM> relate to the matching phase 401B. Blocks <NUM> - <NUM> relate to a feature extraction phase 401C within matching phase 401B, in which color and shape features from the query image are extracted. Blocks <NUM> - <NUM> correlate to blocks <NUM> - <NUM> of indexing phase 401A and are substantially similar to the extraction of color histograms and oriented gradients from registered trademarks as in the indexing phase 401A. In other embodiments, the order of the extraction of the color histogram and the oriented gradients histogram may be reversed or processed simultaneously (in other words, the order of blocks <NUM>/<NUM> and <NUM>/<NUM> may be switched or processed substantially simultaneously). The indexing phase 401A operations are first described below followed by the matching phase 401B operations.

For the indexing phase 401A, at a block <NUM>, the networked system <NUM> (e.g., application servers <NUM>) retrieves a registered trademark image from a database, such as the registered trademark image database <NUM>, which may refer to database <NUM>, a third party database <NUM>, or another database not necessarily housed within device machine <NUM> or application server <NUM> but accessible to device machine <NUM> or application server <NUM>. Database <NUM> includes registered trademark images and information associated with the registered trademarks. Information about a registered trademark includes, but is not limited to, registrant, country, Nice classification, associated dates, description, and the like. The registered trademark image may include various words, letters, colors, lines, shapes, patterns, and/or the like. Although operations taken on a single trademark image is discussed herein, it is understood that a plurality of images can be processed simultaneously with each other in batch jobs. Operations taken on a given single trademark image is for ease of discussion only.

At a block <NUM>, a color histogram is extracted or the color distribution of the image is identified by color histogram module <NUM>. The indexing module <NUM> is configured to appropriately index and add the color histogram image feature data corresponding to the trademark image to the image index database <NUM>. The image can be indexed based on one or more attributes. For example, the image can be indexed according to its extracted color features, and extracted shape features to facilitate rapid look-up of matching items.

The background of the trademark image (e.g., a solid white and/or black background) may be automatically removed. In some embodiments, a sampling mask may provide the spatial sampling area of the inventory image for subsequent feature extraction operations. In other embodiments, a sampling mask is not used or needed as the entire trademark image can be processed. Once the sampling area of the image has been determined-the area within the sampling mask-such sampling area is used for various image feature extraction and identification. In other embodiments, no sampling area is determined as the entire image is processed. Extraction of the color histogram comprises the extraction or identification of the color distribution of the item featured in the image.

The choice of color representation is relevant for extraction of color distribution. Color can be represented in various ways. A color space suited for trademark images is the red, green, and blue (RGB) color space, where all colors are represented with respect to three color channels red, green, and blue in <NUM>-dimensions. In one embodiment, the trademark image (more particularly, within the sampling mask area or the entire image) is sampled or scanned uniformly (also referred to as uniform sampling) using RGB color space. The pixels from the image are sampled. Once these pixels are sampled, information (e.g., image features or attributes) is extracted from each of these pixels. The information can be extracted based on the pixel or the pixel along with its immediate neighbors. Information about color is extracted on a pixel by pixel basis, and then combined, consolidated, or pooled into a collection of values (also referred to as features). In the case of color features, histogram techniques are used-to obtain, for example, a color histogram. A histogram comprises a consolidation of occurrences of various values an item, such as a pixel, can take.

<FIG> illustrates details of blocks <NUM> and <NUM>, processed after blocks <NUM> and <NUM>, respectively, and before blocks <NUM> and <NUM>, respectively, according to some embodiments. As noted above, in other embodiments, the processing order for color histograms and oriented gradients histograms may be switched or substantially simultaneous.

In one embodiment, at block <NUM>, a median filter algorithm is applied to smooth the trademark image before extracting a color histogram. At block <NUM>, the background (e.g., a solid white and/or black background) is removed. At block <NUM>, the entire image (all pixels after background removal) is sampled for each of the RGB color channels. At block <NUM>, the most significant two bits are concatenated from each of the RGB channels to provide or produce <NUM>-bits data for all the pixels of a given image, and then the <NUM>-bits data are used to populate the histogram. At block <NUM>, the <NUM>-bits data are used to extract a <NUM>-bin color histogram for the trademark image.

It is noted that in some embodiments, once the bins are accumulated, weights are not applied, while in other embodiments, weights to the bins may be applied. The non-weighted or weighted adjusted samples are stacked or combined together to generate a resulting stacked histogram corresponding to the three dimensions/channels of the RGB color space. It is noted that RGB values may be normalized to range from <NUM> to <NUM> in some embodiments. It is further noted that the range of valid values for the RGB channels are the same as one another in some embodiments.

Previous methods and systems have used other color spaces, such as HSV, or the full RGB color map, and Euclidean distance space for comparison. However, using original highly segmented RGB values gives higher variation and less accurate results for trademark images, which do not often include a high variation of dominant colors (e.g., often using only tones, degradations, or variations of the same color). Accordingly the present disclosure provides a reduced or decreased color segmentation by grouping similar colors together, thereby allowing for more accurate color matching in trademark images.

Advantageously, the present disclosure provides for smoothing the trademark image to suppress the noisy pixels or to reduce or eliminate the negative effects of noise in the image, removing the background pixels, and extracting the color histogram for the foreground (e.g., by removing white and/or black background). Furthermore, the present disclosure decreases the scale of colors considered (in other words, similar colors are grouped together) as a <NUM>-bin (<NUM> to power of <NUM>) color histogram. Less detail has been discovered to be more advantageous for color matching of trademark images. In addition, the use of Bhattacharyya distance (which is less sensitive to elliptical color distribution and which allows for an elliptical distribution) instead of Euclidean distance (which is more sensitive to elliptical color distribution/which allows for spherical clusters) provides more accurate color matching in trademark images.

Returning to <FIG>, at block <NUM>, an oriented gradients histogram is extracted from the trademark image by oriented gradients histogram module <NUM>. The indexing module <NUM> is configured to appropriately index and add the oriented gradients histogram feature data corresponding to the trademark image to the image index database <NUM>. The image can be indexed based on one or more attributes. For example, the image can be indexed according to its extracted color features (e.g., a color histogram), extracted shape features (e.g., an oriented gradients histogram), associated information (e.g., Nice classification), and the like, to facilitate rapid look-up of matching items.

<FIG> illustrates details of blocks <NUM> and <NUM>, processed after blocks <NUM> and <NUM>, respectively, and before block <NUM>, according to some embodiments. As noted above, in other embodiments, the processing order for color histograms and oriented gradients histograms may be switched or substantially simultaneous. In one embodiment, at block <NUM>, the image is converted to grayscale from color (if in color). Converting to grayscale format, which represents the linear contribution of the three RGB color channels, reduces unnecessary color channels data, and thus allows for greater efficiency and speed in obtaining a shape feature histogram.

At block <NUM>, a median filter is applied to the image to suppress or remove noisy pixels prior to applying an edge detection or shape algorithm.

At block <NUM>, horizontal and vertical derivatives (also referred to as x- and y-derivatives) are calculated (e.g., by Scharr operator) for each pixel of the image. A horizontal derivative provides a horizontal gradient and a vertical derivative provides a vertical gradient.

In one embodiment, a Sobel filter may be applied to obtain an edge map for all the pixels of the image. The Sobel edge map corresponding to the image comprises a faithful line drawing of the edges included in the image with the colors removed. In other embodiments, it is possible to use other edge detection algorithms, such as a Canny edge detector, to obtain a Canny edge map.

At block <NUM>, an orientation angle (also referred to as an oriented gradient) is calculated by taking the arctangent of the vertical derivative divided by the horizontal derivative, calculated for each pixel at block <NUM>.

At block <NUM>, the image is divided into blocks (which can also be referred to as sub-cells or sub-regions). In one example, the entire image is considered a cell, and the cell is divided into <NUM> x <NUM> blocks (or sub-cells or sub-regions). In another example, the image may be divided into <NUM> x <NUM> cells, and <NUM> x <NUM> blocks within each cell (e.g., 6x6 or <NUM> blocks or sub-cells or sub-regions). Orientation calculations are performed for each pixel in each block. Then each pixel grouped in blocks contributes to the orientation histogram. Thus, for an example of one cell, <NUM> x <NUM> blocks, and a <NUM>-bin histogram per block, an <NUM>-bin oriented gradients histogram is provided as a shape feature.

Magnitude calculations is also performed for each pixel in each block and used as a weighting factor for the orientation histogram. A gradient magnitude may be calculated by taking the square root of the sum of the squares of the horizontal and vertical gradients. The gradient magnitude may then be used as part of a weighting factor for the weighted orientation histogram.

At block <NUM>, a <NUM>-bin weighted oriented gradients histogram is extracted for each block by populating the histogram with the calculated orientation angles with weighted values. In one example, the oriented gradients histogram is divided to the following <NUM> bins with weighted values for higher accuracy: <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>. The weight given to an orientation angle may be a ratio or percentage to accurately populate the angle into the bins. The weight is used such that an angle may contribute to neighboring bins which are included in the histogram. For example, if the calculated angle is <NUM>, the angle may contribute equally to bins <NUM> and <NUM>. For example, if the calculated angle is <NUM>, a <NUM>/<NUM> weight may be given to bin <NUM> and a <NUM>/<NUM> weight may be given to bin <NUM>. At block <NUM>, the orientation histogram module <NUM> applies a weight to each orientation angle of a pixel of the edge map, which results in a weighted orientation histogram that accurately fits the orientation angles into <NUM> bins. Then, the orientation histogram module <NUM> sums the bins in the weighted orientation histogram to obtain a shape feature of the trademark image.

Alternatively, as previously mentioned, a gradient magnitude may be calculated by taking the square root of the sum of the squares of the horizontal and vertical gradients. The gradient magnitude may then be used as part of a weighting factor for the weighted orientation histogram.

Advantageously, as described above, a courser or less fine segmentation (i.e., less blocks or sub-regions) than previously used to obtain shape features may be used for trademark images as the trademark is typically centered and the entire image is processed. Furthermore, the combination of the image shape feature processing and the color feature processing with the decreased color space allow for efficient and accurate trademark image processing and matching.

Thus, image features of a given inventory image are extracted/identified and such image features are stored in the image index database <NUM> appropriately indexed to facilitate rapid look-up in response to query images. By repeating blocks <NUM> - <NUM> as many times as needed, all of the registered trademark images (sets, batches) as desired can be similarly indexed.

With the inventory images indexed, <FIG> will now be described in the context of the matching phase 401B. At block <NUM>, the image capture module <NUM> is configured to receive a query image. In one embodiment, the query image is sent by a device machine <NUM> to the networked system <NUM>. A user of the device machine <NUM> (e.g., a smartphone) may take a photograph of a color, shape, pattern, logo, textile, or the like to capture an image of interest using the camera included in the device machine <NUM>. The photograph is transmitted to the networked system <NUM>, to be the query image, for image feature extraction and to return registered trademark images most similar to the query image. In other embodiments, a user may provide a query image file to device machine <NUM>, which then transmits the image file to networked system <NUM> and image capture module <NUM>.

Device machine <NUM> may interface with the networked system <NUM> via a website using a web browser. A query image may be sent to the networked system <NUM> to extract features from the query image. The networked system <NUM> uses the query image's extracted features to find matches with registered trademark images. The top matches are returned to the device machine <NUM> formatted in a match results web page.

Device machine <NUM> may either install an application to interface with networked system <NUM> or access a website hosted by the networked system <NUM>. When a user launches the application at the device machine <NUM>, the application facilitates the user to input or otherwise specify a query image. As an example, the application may include camera capabilities (or otherwise access a separate camera app) to permit the user to take a photo or otherwise obtain an image of interest (e.g., shape or word that is colored and/or patterned). Furthermore, the application may upload or link to a digital file of the query image already on device machine <NUM>.

For blocks <NUM> - <NUM>, the same operations as described above for blocks <NUM> - <NUM> are performed except the operations are taken on the query image instead of a registered trademark image.

For block <NUM>, the comparison module <NUM> is configured to compare the color histogram and oriented gradients histogram of the query image to the color histograms and oriented gradients histograms of the registered trademark images to find one or more registered trademark images similar to or matching the query image.

<FIG> illustrates further details of block <NUM>, processed after blocks <NUM> and <NUM>, and before block <NUM>, according to some embodiments. Comparison module <NUM> performs two comparisons for each pair of a query image and an indexed trademark image: a comparison of the color histograms and also a comparison of the oriented gradients histograms. At block <NUM>, mathematical distances between a color histogram of the query image and a color histogram of each registered trademark image is calculated. At block <NUM>, mathematical distances between an oriented gradients histogram of the query image and an oriented gradients histogram of each registered trademark image is calculated. As noted above, in other embodiments, the processing order for comparing color histograms and oriented gradients histograms may be switched or substantially simultaneous.

In both algorithms explained above (i.e., color and shape based algorithms), a similarity measure for color and shape features between a given query/input image and all the database images listed in a folder is calculated and sorted, for example according to Bhattacharyya distance. The best matched image has the smallest distance similarity measure. Advantageously, the present method provides for fast and accurate trademark image comparison and retrieval as image texture is not considered. Image texture is not considered as most trademark images do not have texture or many color transitions. Accordingly, less data processing is needed for trademark images, and thus more efficient but still accurate image comparison and retrieval are made possible by the present invention that utilizes less than full RGB color space for extracting the color feature, and courser segmentation (e.g., blocking) for extracting the shape feature. Surprisingly, in extracting both the color features and the shape features, courser and reduced binning is utilized to increase both processing efficiency and accuracy of image comparison for large sets of trademark-type images (on the order of thousands to millions).

A similarity score or value can be assigned for each comparison of the (color or orientation) histograms. For each image pair, the final similarity score is the sum of the color histogram comparison similarity score and the orientation histogram comparison similarity score.

Possible similarity schemes include, but are not limited to, Bhattacharyya distance, cosine similarity (inner product) correlation, Chi-squared correlation, and intersection. In one embodiment, the similarity value or score is calculated to be the Bhattacharyya distance. The Bhattacharyya distance comprises the square root of the complement of cosine similarity of the square root of the color histograms or oriented gradients histograms, as a function of the number of bins.

Returning to <FIG>, at block <NUM>, the image retrieval module <NUM> provides the similar or matching results to the device machine <NUM> for display on the screen of the device machine <NUM>. The matching results (e.g., registered trademark images and possibly associated item information) are displayed in order of degree of similarity to the query image. In other words, a similarity value is assigned for each pair of the query image and an indexed trademark image. The similarity scores are sorted so that matching results can be displayed on the device machine <NUM> in the order of similarity to the input query image in some embodiments.

In some embodiments, the system or the user may set an upper limit on the number of matching results that may be presented in response to the query image. Further, in some embodiments, a threshold value for the similarity measure may be set such that images not meeting the threshold value are not retrieved or displayed.

Referring now to <FIG>, <FIG> illustrates an example screen shot or user interface screen <NUM> including a query image <NUM>, <FIG> illustrates high-level block diagrams of a color histogram extraction, <FIG> illustrate high-level block diagrams of an oriented gradients histogram extraction, and <FIG> illustrates an example screen shot or user interface screen <NUM> showing retrieved or returned search results after comparison to a query image. <FIG> and <FIG> illustrate various user interface (UI) screens displayed on the device machine <NUM> pertaining to the matching phase according to some embodiments.

In user interface <NUM> of <FIG>, the example query image <NUM> shown on the screen comprises letters, colors, and shapes. The application on device machine <NUM> (application to interface with networked system <NUM> or access a website hosted by the networked system <NUM>) transmits the query image <NUM> to the networked system <NUM>, and in response, the networked system <NUM> performs image feature extraction operations and provides matching results as discussed above. User interface <NUM> further includes, in an embodiment, search options <NUM> for describing the query image, including but not limited to the following: image type (Marka Tipi) such as a word or letter, a shape, or both; color type (Renk) such as black and white or color; and class (siniflar) or description such as a Nice classification or other numbered description. In an embodiment, user interface <NUM> further includes data input menu <NUM>, such as but not limited to trademark image name (Marka Adi), keyword (Anahtar Adi), an application number (Basvuru No.), a file number (Bulten No.), and an option for loading (Yukle) or editing (Resmi duzenle) a query image.

<FIG> illustrates an example implementation of blocks <NUM> or <NUM> for extraction of an image color feature, as illustrated by a portion of a color histogram <NUM> corresponding to a trademark image with color. The two most significant bits <NUM> (e.g., <NUM>, <NUM>, and <NUM>) of each RGB channel <NUM> are taken and combined (e.g., concatenated) to create a new <NUM>-bit value <NUM> for each pixel in a given image.

The histogram <NUM> of all <NUM>-bit values of each pixel is then calculated or extracted and combined in a <NUM>-bin color histogram. The horizontal axis represents the bins (a total of <NUM> bins) for R, G, and B channels <NUM> for each image pixel. The bins correspond to different colors in the RGB color space. The vertical axis represents the amount of the bins represented on the horizontal axis. Thus, high peaks represent colors that are more prominently present than other colors for the multi-colored trademark image.

A stacked 1D histogram presents information about the image in a relatively compact form, instead of storing values of all pixels of the sampled portion of the image. The vertical axis represents the number of pixels that take the corresponding value on the horizontal axis. Hence, the resulting stacked 1D histogram identifies the colors present within the entire trademark image (or within the sampling mask of the image).

<FIG> illustrate an example implementation of blocks <NUM> or <NUM> for extraction of image shape features. A trademark image <NUM> may include letters, shapes, and colors. Image <NUM> is divided into blocks <NUM>, such as in a <NUM> x <NUM> configuration resulting in nine blocks <NUM> (also called sub-cells or sub-regions). Then a <NUM>-bin oriented gradients histogram <NUM> is extracted for each of the nine blocks <NUM>, as described above with respect to <FIG> (system and apparatus for shape histogram extraction) and <FIG> and <FIG> (methods of shape histogram extraction). Each block <NUM> has a D vector length. The distance Dt between each histogram is measured by using a distance measurement algorithm, as shown at <NUM>.

The shape histogram method provides one cell with <NUM> x <NUM> blocks <NUM> for an image. Each block <NUM> provides <NUM> bins for a total of <NUM> bins for each image. Most of the trademark images are localized to the center of the image so calculating upper-left, upper-right, bottom-left and bottom-right sides is time consuming and may not contribute to the histogram as well. In comparison to an image which is divided into <NUM> x <NUM> cells, each cell having <NUM> x <NUM> blocks and each block having <NUM> bins, which means <NUM> x <NUM> x <NUM> x <NUM> x <NUM> or <NUM> bins or features. Calculating the similarity between each vector which has <NUM> elements takes much more time than a vector which has <NUM> elements.

For each pixel of the image <NUM>, a matrix <NUM> representing shape elements is calculated, and then vertical derivatives Gx and horizontal derivatives Gy are calculated. The vertical derivative Gx divided by the horizontal derivative Gy results in a derivative quotient. An orientation angle θ is then calculated by taking the arctangent of the derivative quotient (vertical derivative divided by the horizontal derivative). A gradient magnitude may be calculated by taking the square root of the sum of the squares of the vertical derivative and the horizontal derivative (square root of (Gx<NUM> + Gy<NUM>)). Graph <NUM> illustrates the vertical derivative Gx, the horizontal derivative Gy, the magnitude M, and orientation angle <NUM>, in one example.

In one example, oriented gradients histogram <NUM> is populated by weighted orientation angles θ using <NUM> bins: <NUM>-<NUM> degrees, <NUM>-<NUM> degrees, <NUM>-<NUM> degrees, <NUM>-<NUM> degrees, <NUM>-<NUM> degrees, <NUM>-<NUM> degrees, <NUM>-<NUM> degrees, <NUM>-<NUM> degrees, and <NUM>-<NUM> degrees. In another example, oriented gradients histogram <NUM> may be populated by weighted orientation angles θ including a magnitude component.

In <FIG>, screen <NUM> shows returned or matching results displayed on the device machine <NUM>. All items that are deemed to "match" the query image are displayed. The matching results may be organized and displayed in various ways, such as based upon similarity value (e.g., by smallest Bhattacharyya distance), such as total similarity value, color histogram similarity value, or oriented gradients histogram similarity value. Screen <NUM> shows order numbers <NUM> for the returned images based on Bhattacharyya distance, with the smallest Bhattacharyya distance having the smallest order number and the corresponding retrieved image appearing earlier on the list of retrieved results. The matching results may also be organized and displayed based upon a combination of similarity value and categories, such as based upon Nice classification, registrant, or the like. The matching items within a selected category may be then ordered from highest to lowest similarity score for that category. For each matching image, information such as, but not limited to, a color image and one or more of a similarity score, registrant, owner, and country may be displayed.

The user can select from among the displayed matching images, for example, a third ranked image. In response, additional image details may be provided about the selected image. The user can navigate within the matching results to view one or more registered trademarks of interest. Thus, in one embodiment, a user can simply provide a query image, either by pointing to an image file or taking a photo of something, and the application in conjunction with the networked system <NUM> may automatically return images from a database of registered trademarks that are similar or match the color and shape.

When the set of registered trademarks includes tens of thousands to millions of registered trademarks, the number of matching results for a given query image can be prohibitively high. Especially when the matching results are viewed on smaller displays, as is common in smartphones and other mobile devices, viewing exhaustion can occur well before all of the matching items are viewed by the user. Even on larger displays, users are more likely to view top matching results than (much) lower matching results. Thus, efficient and accurate results are highly desirable.

In order to provide more user options in viewing retrieved images and possibly improve desired results, user indications during interaction with the provided matching results can be used to re-rank or re-order the registered trademarks within the initial matching results to better suit the user's interest or to improve results.

For example, during user interaction with the matching results corresponding to a given query image at the device machine <NUM>, the user may indicate an interest in a particular registered trademark from among the match results. Image details corresponding to the selected image may be displayed (e.g., a larger image, additional images, registrant, country, and/or the like).

A button or other indicator may give the user an option to request re-ordering based upon one or more correlation, association, or recommendation rules. Several factors, such as the user selection of a particular trademark, trademarks, content of the selected trademark(s) (e.g., particular histogram type to prefer - either color or shape features), or trademark-associated information such as Nice classification, registrant, owner, country, and/or the like, may be used for the correlation or association to other trademark images for the re-ordering/re-ranking process. The user interest detection module <NUM> at the networked system <NUM> receives the user indication/preference for the particular image or associated information and may then cause a re-ordering of the initially matched trademark images based upon correlation or association rules, such as the user selected trademark or associated information. A user preference for more than one image within the match results can be indicated prior to initiation of re-ordering or re-ranking. Then re-ordered match results may be displayed on the device machine <NUM>. The re-ordered matching results comprise refinement of the initial matching results in accordance with additional user input regarding image(s) of interest within the initial matching results.

The re-ordering/re-ranking operation can be repeated more than once for a given matching results in response to new or additional preference for certain image(s) or image-associated information within the matching results provided by the user.

Thus, the present invention provides for the capture of a query image and image features extraction, presentation of registered trademark images that have the closest match to the image features of the query image, and viewing of registered trademark image details with an option to refine the returned results based upon user interest or selected information.

An alternative way to obtain the query image, other than the user providing an image file or taking a photo and uploading it to the networked system <NUM> via an application or website on the device machine <NUM>, is from websites not associated with the network system <NUM>, such as any website comprising HTML-based web pages. Such an option may be useful to search for registered trademarks by country category.

User interest detection for re-ordering or refining the match results and acquisition of query images from third party websites are facilitated by a browser plugin installed on the device machine <NUM>, as further described hereafter. A browser plugin provided by the networked system <NUM> (e.g., application servers <NUM>) is installed on the device machine <NUM>. The browser plugin comprises a browser extension, jquery snippet of code, or browser plugin. The browser plugin can be a standalone module or part of an application. The browser plugin includes at least the browser plugin module <NUM>.

The browser plugin module <NUM> is configured to monitor web browser activity at the device machine <NUM> to detect a request for a certain website (or web page). The request comprises user entry of a uniform resource locator (URL) address in a web browser included in the device machine <NUM>, or the user clicking on a hyperlink to a web page. The certain website comprises a website (or web page) from among a plurality of websites (e.g., any website comprising HTML-based web pages) from which a query image can be sourced. Such a website (or web page) may also be referred to as a query image source or external third party query image source.

The browser plugin module <NUM> and/or the user interest detection module <NUM> is configured to detect user interest in or attention on an image included in a web page. In one embodiment, user interest in an image is detected when the user hovers a pointing device (e.g., mouse, trackpad, trackball, finger, etc.) at or near a particular image for at least a minimum time period. In alternative embodiments, the browser plugin module <NUM> can provide one or more graphical user interface (GUI) tools for the user to explicitly specify interest in a given image included on the web page. Example GUI tools include, but are not limited to, a pointing tool, a highlighting tool, an area indictor tool, and the like. Whatever the particular way to detect user interest in an image, the detection comprises identifying the user's spatial attention to a specific portion of the displayed web page. Once a user interested image is detected, the browser plugin module <NUM> receives the user interested image, which then comprises the query image that may be automatically applied to the matching phase 401B as described above.

In this manner, image feature data extraction and uses of the image feature data are disclosed herein. In an indexing phase, registered trademark images are accessed and indexing is performed on the trademark images to extract their image features and attributes, and populate an image index database in which the trademark images are indexed by their image features and item categories. Extracted image features include a color histogram, and an oriented gradients histogram. In a search phase (also referred to as a matching phase), the information in the image index database is accessed when a non-inventory image is received (also referred to as a query image) in order to provide matches or most similar registered trademark images corresponding to the query image. The query image may be provided by a user uploading, pointing to, emailing, messaging, or otherwise transferring or providing an image file. In some embodiments, a photo of an image, or an image included in a (non-affiliated) website or web page comprises an input or query image. Search results comprise registered trademarks that best match the query image.

In another embodiment, user selection or preference for certain of the trademarks or associated information provided in the search results is used to re-order or re-rank the listing order of the trademarks within the results. User's interest in an image included in a website/web page may also be used as the input, sample, or query image to return search results corresponding to that image.

It is contemplated that alternative embodiments for uses of the image feature data, feature extraction, and performing matching are possible. For example:.

Thus, the present invention provides a highly improved trademark image search and retrieval method, system, and computer product for automatically extracting feature data, comparing features and images, retrieving images and associated data, etc., with greatly improved speed, efficiency, and accuracy. Surprisingly, in extracting both the color features and the shape features, the present invention utilizes filtering, data reduction, courser segmentation, and reduced binning to increase both processing efficiency and accuracy of image comparison for large sets of trademark-type images (on the order of thousands to millions).

<FIG> shows a diagrammatic representation of a machine in the example form of a computer system <NUM> within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, is executed. The computer system <NUM> comprises, for example, any of the device machine <NUM>, applications servers <NUM>, API server <NUM>, web server <NUM>, database servers <NUM>, or third party server <NUM>. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a device machine in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a server computer, a client computer, a personal computer (PC), a tablet, a set-top box (STB), a Personal Digital Assistant (PDA), a smart phone, a cellular telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

The example computer system <NUM> includes a processor <NUM> (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory <NUM> and a static memory <NUM>, which communicate with each other via a bus <NUM>. The computer system <NUM> may further include a video display unit <NUM> (e.g., liquid crystal display (LCD), organic light emitting diode (OLED), touch screen, or a cathode ray tube (CRT)). The computer system <NUM> also includes an alphanumeric input device <NUM> (e.g., a physical or virtual keyboard), a cursor control device <NUM> (e.g., a mouse, a touch screen, a touchpad, a trackball, a trackpad), a disk drive unit <NUM>, a signal generation device <NUM> (e.g., a speaker) and a network interface device <NUM>.

The disk drive unit <NUM> includes a machine-readable medium <NUM> on which is stored one or more sets of instructions <NUM> (e.g., software) embodying any one or more of the methodologies or functions described herein. The instructions <NUM> may also reside, completely or at least partially, within the main memory <NUM> and/or within the processor <NUM> during execution thereof by the computer system <NUM>, the main memory <NUM> and the processor <NUM> also constituting machine-readable media.

The instructions <NUM> may further be transmitted or received over a network <NUM> via the network interface device <NUM>.

While the machine-readable medium <NUM> is shown in an example embodiment to be a single medium, the term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term "machine-readable medium" shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present invention. The term "machine-readable medium" shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic media, and carrier wave signals.

It will be appreciated that, for clarity purposes, the above description describes some embodiments with reference to different functional units or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, processors or domains may be used without detracting from the invention. For example, functionality illustrated to be performed by separate processors or controllers may be performed by the same processor or controller. Hence, references to specific functional units are only to be seen as references to suitable means for providing the described functionality, rather than indicative of a strict logical or physical structure or organization.

Certain embodiments described herein may be implemented as logic or a number of modules, engines, components, or mechanisms. A module, engine, logic, component, or mechanism (collectively referred to as a "module") may be a tangible unit capable of performing certain operations and configured or arranged in a certain manner. In certain example embodiments, one or more computer systems (e.g., a standalone, client, or server computer system) or one or more components of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) or firmware (note that software and firmware can generally be used interchangeably herein as is known by a skilled artisan) as a module that operates to perform certain operations described herein.

In various embodiments, a module may be implemented electronically. For example, a module may comprise dedicated circuitry or logic that is permanently configured (e.g., within a special-purpose processor, application specific integrated circuit (ASIC), or array) to perform certain operations. A module may also comprise programmable logic or circuitry (e.g., as encompassed within a general-purpose processor or other programmable processor) that is temporarily configured by software or firmware to perform certain operations.

Accordingly, the term "module" should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), non-transitory, or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. Considering embodiments in which modules or components are temporarily configured (e.g., programmed), each of the modules or components need not be configured or instantiated at any one instance in time. For example, where the modules or components comprise a general-purpose processor configured using software, the general-purpose processor may be configured as respective different modules at different times. Software may accordingly configure the processor to constitute a particular module at one instance of time and to constitute a different module at a different instance of time.

Modules can provide information to, and receive information from, other modules. Accordingly, the described modules may be regarded as being communicatively coupled. Where multiples of such modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) that connect the modules. In embodiments in which multiple modules are configured or instantiated at different times, communications between such modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple modules have access. For example, one module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further module may then, at a later time, access the memory device to retrieve and process the stored output. Modules may also initiate communications with input or output devices and can operate on a resource (e.g., a collection of information).

The Abstract is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter may lie in less than all features of a single disclosed embodiment.

Embodiments of the present invention may be embodied as a system, method, or computer program product (e.g., embodiments directed toward an image searching system, method, or computer program product). Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit", "module", or "system". For example, an image searching method may be embodied in a software and hardware system that can be housed in a portable device such as a tablet, laptop, camera, phone, and the like. In another example, a client and server computer in operable communication and combination, may be in its entirety said to be embodied in a system. Furthermore, aspects of the present embodiments of the disclosure may take the form of a computer program product embodied in one or more computer readable medium/media having computer readable program code embodied thereon. Methods may be implemented in a special-purpose computer or a suitably programmed general-purpose computer.

Any combination of one or more computer readable medium/media may be utilized.

Computer program code for carrying out operations for aspects of the present embodiments of the disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages.

Aspects of the present embodiments of the disclosure are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention (e.g., <FIG>).

Claim 1:
A method for content based search and retrieval of trademark images, the method comprising:
extracting color features from a plurality of trademark images and a query trademark image by generating (<NUM>), with a processor, a non-weighted <NUM>-bin color histogram for each trademark image using (<NUM>) <NUM>-bit color data for each pixel of the each trademark image, wherein the <NUM>-bit color data includes <NUM> most significant bits from each of a red, green, and blue channel for each pixel of the each trademark image to provide a reduced color segmentation;
extracting shape features from the plurality of trademark images and the query trademark image by dividing (<NUM>) each trademark image into <NUM> x <NUM> blocks; and
extracting (<NUM>) a <NUM>-bin shape histogram for each block of each trademark image using orientation angle data with weighted values to obtain a shape histogram of each trademark image; and
generating, by the processor, distance similarity measures on the color features and the shape features of the query trademark image with the plurality of trademark images;
retrieving and displaying trademark images on a display in an order according to distance similarity measures.