Tag-based font recognition by utilizing an implicit font classification attention neural network

The present disclosure relates to a tag-based font recognition system that utilizes a multi-learning framework to develop and improve tag-based font recognition using deep learning neural networks. In particular, the tag-based font recognition system jointly trains a font tag recognition neural network with an implicit font classification attention model to generate font tag probability vectors that are enhanced by implicit font classification information. Indeed, the font recognition system weights the hidden layers of the font tag recognition neural network with implicit font information to improve the accuracy and predictability of the font tag recognition neural network, which results in improved retrieval of fonts in response to a font tag query. Accordingly, using the enhanced tag probability vectors, the tag-based font recognition system can accurately identify and recommend one or more fonts in response to a font tag query.

BACKGROUND

Recent years have seen a proliferation in the use of computing devices in the area of digital typography with respect to creating and editing electronic documents. Indeed, it is now commonplace for individuals and businesses to use digital typography to create customized web pages, e-mails, magazines, marketing materials, and other electronic documents utilizing desktop and laptop computers, mobile devices, tablets, smartphones, or other computing devices.

Recent years have also seen an increase in the type and variety of digital fonts utilized in electronic documents. Individuals can find, access, and install digital fonts on a computing device for use in creating electronic documents from large repositories. For example, an electronic document can use digital fonts selected from a collection of thousands of digital fonts.

A significant challenge that has arisen with the increase in the number of digital fonts is the capability to efficiently find a desired font or font style. One type of font retrieval is tag-based font retrieval in which a user provides a query term to search for corresponding fonts. A number of problems have made developing a tag-based font search system challenging. For example, font tags are not standardized. To illustrate, the number of potential font tags is unlimited. As part of this issue, a font tag can describe different categories of a font, such as the font's visual appearance, characteristics, usage, classification, family, mood, special properties, and/or other attribute categories. Accordingly, the vastness of a font tag library creates difficulty in training a tag-based font search system. Indeed, a large number of font-tags can lead to misclassification and inaccurate results. Further, this problem is exacerbated as new fonts, with or without font tags, are constantly being created.

In addition, font tags are subjective to users creating them. The importance and informativeness of tags may vary from user to user. In addition, there is also a large gap between the semantics of natural language in a font tag and the visual appearance of the font. Further, the tag description from a user can sometimes be ambiguous and vague. Also, different users may choose vastly different tags when describing fonts of similar visual appearance in their minds. For example, different users can use the same tag to describe different looking fonts. Similarly, different users can use unrelated tags to describe the same font. Thus, relying on user tags alone leads to inconsistent and unreliable results.

Because of these issues, attempts at creating tag-based font search systems have been unsuccessful and resulted in unsatisfactory systems. For instance, these tag-based font search systems necessitate large amounts of memory and computational requirements. Furthermore, conventional tag-based font search systems are inaccurate due to training using user tags. Additionally, conventional tag-based font search systems typically have relatively small numbers of tags and fonts, meaning the system are limited and inflexible.

These and other problems exist with regard to retrieving digital fonts utilizing a tag-based font search system using existing systems and methods.

SUMMARY

Embodiments of the present disclosure provide benefits and/or solve one or more of the foregoing or other problems in the art with systems, computer media, and methods for effectively recognizing digital fonts (or simply “fonts”) based on font tag queries. For example, the disclosed systems utilize deep learning neural networks to identify fonts in response to font tag queries. More particularly, the disclosed systems use a combination of a font tag recognition neural network and a font classification neural network to generate font tag probability vectors that indicate the probability that a given font is associated with various tags. Specifically, the disclosed systems weight deep features determined by the font tag recognition neural network with implicit font classification information from the font classification neural network to improve the accuracy of font tag probability vectors. Then given a font tag query, the disclosed systems use the font tag probability vectors to identify fonts associated with the font tag query.

To illustrate, the disclosed systems identify a set of font images that are labeled with font tags and font classifications. Using the font images and tags, the disclosed systems train a font tag recognition neural network to determine font tag probabilities that each font corresponds to each font tag. In addition, the disclosed systems train an implicit font classification attention model that transforms the output of a trained font classification neural network into a font classification attention map matching the dimensions of hidden features of the font tag recognition neural network. Further, the disclosed systems jointly train portions of the font tag recognition neural network with the implicit font classification attention model to generate enhanced font tag probabilities that each font corresponds to each font tag. Using the generated enhanced font tag probabilities, the disclosed systems can accurately retrieve and recommend fonts in response to a font tag query.

The following description sets forth additional features and advantages of one or more embodiments of the disclosed systems, computer media, and methods. In some cases, such features and advantages will be obvious to a skilled artisan from the description or may be learned by the practice of the disclosed embodiments.

DETAILED DESCRIPTION

This disclosure describes one or more embodiments of a tag-based font recognition system that utilizes a multi-learning framework to develop and improve tag-based font recognition using deep learning neural networks. In particular, the tag-based font recognition system jointly trains a font tag recognition neural network with a font classification attention model to generate font tag probability vectors that are enhanced by implicit font classification information. Further, using the enhanced tag probability vectors, the tag-based font recognition system can identify one or more fonts to recommend in response to a font tag query.

To illustrate, in one or more embodiments, the tag-based font recognition system (or simply “font recognition system”) trains a font tag recognition neural network to determine initial font tag probabilities that indicate the likelihood that each font in a set of fonts corresponds to each font tag in a set of font tags. In addition, the font recognition system trains a font classification attention model to generate a font classification attention map that transforms font classification feature vectors outputted from a separately trained font classification neural network to match the dimensions of feature vectors outputted from the trained font tag recognition neural network. Further, the font recognition system jointly trains the font tag recognition neural network with the implicit font classification attention model using the font classification attention map and the font classification feature vectors to determine an enhanced tag probability for each font in the set of fonts.

In various embodiments, the font recognition system utilizes the enhanced tag probabilities to retrieve fonts from the set of fonts in response to a font tag query. More particularly, the font recognition system receives a font tag query that includes a font tag (or multiple tags). Utilizing the font tag, the font recognition system determines the tag probabilities between the font tag and each of the fonts in the set of fonts from the enhanced tag probabilities. Further, the tag-based font recognition system can then return one or more fonts having the highest (e.g., most favorable) probabilities in response to the font tag query.

As mentioned above, the font recognition system trains multiple deep learning neural networks. In one or more embodiments, the font recognition system initially trains the font classification neural network. For example, the font recognition system generates a first convolutional neural network (CNN) and trains it based on font images and corresponding font tags to determine an initial font tag probability of each font of a set of fonts.

In various embodiments, the font recognition system also trains a font classification neural network. For example, the font recognition system generates a second CNN and trains it to predict font classifications given an unclassified input font image. Also, the font recognition system can train the font classification neural network based on the same font images but with corresponding font classification labels as ground truths rather than font tags. Alternatively, the font recognition system utilizes a pre-trained font classification neural network.

Utilizing the trained font classification neural network, in one or more embodiments, the font recognition system generates and trains an implicit font classification attention model. In some embodiments, the implicit font classification attention model is added to the trained font classification neural network and further trained to generate a font classification attention map. As mentioned above, the font classification attention map transforms the font classification feature vectors from the font classification neural network to align with feature vectors of the font tag recognition neural network.

More particularly, in various embodiments, the font tag recognition neural network generates font tag recognition feature vectors having n-dimensions that correspond to the number of font tags used in training. Similarly, the font classification neural network generates font classification feature vectors having m-dimensions that correspond to the number of font classifications used in training. Thus, the implicit font classification attention model generates a font classification attention map for each font that converts the m-dimensions of the font classification feature vectors into an n-dimension mapping for each font (i.e., a font classification attention map). Using the font classification attention maps, the font recognition system can inject font classification information from the font classification neural network into the font tag recognition neural network. Indeed, the font recognition system can weight (e.g., using element-wise multiplication) hidden layers of the font tag recognition neural network with implicit deep font classification information to improve font tag predictions.

As mentioned above, the font recognition system can jointly train the font tag recognition neural network and the implicit font classification attention model. In one or more embodiments, the font recognition system jointly trains the higher fully-connected neural network layers of the font tag recognition neural network with the implicit font classification attention model utilizing font tag recognition feature vectors weighted by corresponding font classification attention maps. For example, the font recognition system measures and back propagates error loss to the higher fully-connected neural network layers of the font tag recognition neural network with the implicit font classification attention model while not tuning the pre-trained lower neural network layers of the font tag recognition neural network or tuning the remaining layers of the pre-trained font classification neural network.

In some embodiments, the font recognition system can increase the robustness of the font classification attention map before injecting the font classification attention map into the hidden layers of the font tag recognition neural network. For example, upon receiving an image of a font at the implicit font classification attention model, the font recognition system creates additional images of the font that include different characters of the font chosen at random. The font recognition system then feeds the additional images into the implicit font classification attention model to obtain different font classification attention values for the font, which are averaged together to form a more robust font classification attention map for the font.

As previously mentioned, the font recognition system provides numerous advantages and benefits over conventional systems and methods. As an example, the font recognition system utilizes deep learning neural networks to learn, predict, and retrieve fonts based on font tag queries. Indeed, the font recognition system generates and jointly trains multiple neural networks and models to improve the accuracy of tag-based font retrieval.

As another example, in many embodiments, the font recognition system can efficiently analyze heterogeneous (e.g., non-standardized, subjective, vague, and/or ambiguous) font tags and generate uniform font-tag prediction representations. Indeed, the font recognition system learns objective, well-defined, and precise metrics between font tags and font classifications (e.g., visual features) for fonts through injecting implicit font classification information into the font tag recognition neural network. In this manner, as new fonts are created, the font recognition system can accurately and efficiently learn correspondences between fonts and any number of font tags.

Because the user embeddings system efficiently processes non-standardized font tags, the font recognition system provides increased flexibility over conventional systems. Indeed, the ability of the font recognition system to efficiently treat non-standardized font tags enables the font recognition system to operate given the many different font tag categories (e.g., the font's visual appearance, characteristics, usage, classification, family, mood, special properties, and/or other attribute categories).

The font recognition system also improves computer efficiency. Indeed, by more accurately and precisely identifying relationships between fonts, font tags, and font classifications, the font recognition system can reduce computing resources required to generate, predict, and retrieve fonts, especially in response to font tag queries. Additionally, as described below in connection withFIG. 7, researchers compared embodiments of the font recognition system to baseline font tag recognition neural networks and found that the font recognition system outperformed these baseline neural networks.

Moreover, the font recognition system improves efficiency and maximizes the processing capabilities of current mobile client devices, enabling mobile client devices to achieve results not previously possible. Indeed, in some embodiments, the font recognition system can generate a font/font tag correlation database that is compact in size and can be easily downloaded, stored, and executed on client devices. Accordingly, the font recognition system enables mobile client devices to locally, easily, and quickly identify fonts based on font tags from font tag queries.

Additional advantages and benefits of the font recognition system will become apparent in view of the following description. Further, as illustrated by the foregoing discussion, the present disclosure utilizes a variety of terms to describe features and advantages of the font recognition system. For example, as used herein, the term “digital font” (or simply “font”) refers to a defined set of digital characters (e.g., glyphs). In particular, the term “font” includes a collection of digital characters of a particular style or typeface. A font includes digital files with the extensions, such as, but not limited to: .ttf, .otf, .fnt, .abf, .ttc, .suit, .woff, .vnf, .t65, .sfp, .sad, .pmt, .pfm, .pfb, .pfa, .odtff, .mf, .gdr, .fon, .fnt, .font, .etx, .eot, .compositefont, .acfm, .afm, or .amfm. For example, the term digital font includes fonts entitled Times New Roman, Helvetica, Arial, PT Sans Regular, Corbel, or other font titles.

As used herein, the term “font image” refers to any type of electronic document, image, or file that includes written text (i.e., glyph content) in a font. For example, a font image can include an electronic document in the form of an image file (permanent or temporary), a clipboard file, a word-processing document, a portable document file (PDF), an e-mail file, a text file, a web page, or any other electronic file. The font recognition system can utilize font images to train one or more neural network (e.g., font training images). In addition, a font image can be associated with metadata, such as fonts (e.g., font names), font tags, and/or font classifications that provide information about the font used in the font image.

The term “font tag,” as used herein, refers to a label that describes a characteristic or attribute associated with the font. In particular, the term “font tag” can describe the look, style, or feel of a font. In some embodiments, font tags are divided into categories, such as visual appearance, characteristics (e.g. thin, block, modern, antique, crazy), usage or utility (e.g. headline, poster, magazine, logo), family, mood, special properties, and/or other attribute categories (e.g., ornamental, kid, ancient). Additionally, a font tag can also indicate attributes corresponding to a font, such as style (e.g., regular, bold, italic, shadowed, underlined, strikethrough, hand-written, display, subscript, or superscript), weights (e.g., light, regular, and heavy weights), widths (e.g., condensed, regular, and extended widths), capitalization styles (e.g., all caps, small caps, title case, and sentence case), x-heights (e.g., small, regular and large x-heights), and contrasts (e.g., low, regular, and high contrasts).

Similarly, the term “font classification” refers to a font family, category, and/or font name and can include pre-defined categories utilized to classify digital fonts. For instance, font classifications include font classes (i.e., Serif, Sans Serif, Slab Serif, Script, Blackletter, Mono, Hand, or Decorative). In some embodiments, a font tag and a font classification include overlapping labels or information. For example, a category of font tags can include font classifications.

As mentioned above, the font recognition system utilizes machine learning and various neural networks in various embodiments. Indeed, the font recognition system trains various neural networks to generate font feature vectors including font tag recognition feature vectors and font classification feature vectors. The term “machine learning,” as used herein, refers to the process of constructing and implementing algorithms that can learn from and make predictions on data. In general, machine learning may operate by building models from example inputs, such as a training set of font images corresponding to a plurality of fonts, to make data-driven predictions or decisions. Machine learning can include neural networks and/or machine-learning models (e.g., the font tag recognition neural network, the font classification neural network with or without the implicit font classification attention model, a generative adversarial network (“GAN”) having a generator neural network and a discriminator neural network, and a retrieval model).

As used herein, the term “neural network” refers to a machine learning model that can be tuned (e.g., trained) based on inputs to approximate unknown functions. In particular, the term neural network can include a model of interconnected neurons that communicate and learn to approximate complex functions and generate outputs based on a plurality of inputs provided to the model. For instance, the term neural network includes an algorithm (or set of algorithms) that implements deep learning techniques that utilize a set of algorithms to model high-level abstractions in data using supervisory data to tune parameters of the neural network.

In addition, in one or more embodiments, the term neural network can include deep convolutional neural networks (i.e., “CNNs”), or other types of deep neural networks. The description and figures below generally refer to a CNN, which includes lower layers (e.g., convolutional, deconvolutional, and pooling layers), and higher layers (e.g., fully-connected layers and classifiers). Example architecture of a CNN is provided inFIG. 2C.

As used herein, the term “loss function” or “loss model” refers to a function that indicates error loss between feature vectors and/or probability vectors in multi-dimensional vector space. A machine-learning algorithm (e.g., neural network) can repetitively train to minimize and/or maximize font classification error loss (e.g., font classification error loss or tag-based font error loss) based on ground truths (e.g., font classifications or font tags). Indeed, the loss function provides feedback, which is back propagated, to one or more layers of a neural network to tune/fine-tune those layers. Examples of loss functions include a sigmoid unit function, a softmax classifier with cross-entropy loss, a residual loss function, a perceptual loss function, a total variance loss function, a texture loss function, a hinge loss function, and a least squares loss function.

As used herein, joint training (or joint learning) refers to tuning parameters of multiple learning models together. In particular, joint training (or learning) includes solving a plurality of learning tasks at the same time while utilizing the roles and constraints across the tasks. For example, the font recognition system can employ joint learning to iteratively or simultaneously train and tune weights and parameters of the various neural networks and/or machine-learning models. In some embodiments, joint training includes alternating training back and forth between the font tag recognition neural network and the implicit font classification attention model and/or changing the learning rates, as described further below.

As mentioned above, the font recognition system generates font feature vectors such as a font classification feature vector and a font tag recognition feature vector. As used herein, the term “font feature vector” (or simply “feature vector”) refers to a vector of numeric values representing characteristics and/or attributes of a font. In particular, the term “font feature vector” includes a set of values corresponding to latent and/or patent attributes and characteristics of a font. In one or more embodiments, a feature vector is a multi-dimensional dataset that represents a font.

In one or more embodiments, a feature vector includes a set of numeric metrics learned by a machine-learning algorithm such as a neural network. For example, a font classification feature vector can include font glyph data, such as glyph curvature, glyph spacing, glyph size, glyph shape, glyph width, glyph height, glyph location (e.g., glyph location in relation to a baseline), glyph area, glyph orientation, number of curves per glyph, arc length, glyph contrast, and font classification features (e.g., font features utilized to classify a font) in addition to hidden or latent features. In addition, a font classification feature vector can provide numeric values for the font class of a font. Further, the font classification feature vector can be m-dimensions based on the number of font classifications corresponding to a font training image set.

Similarly, a font tag recognition feature vector can include font information indicating characteristics or attributes associated with the font as indicated by corresponding font tags. Indeed, a font tag recognition feature vector can provide numeric values for the font tags associated with a font. In addition, the font tag recognition feature vector can be n-dimensions based on the number of font tags corresponding to a font training image set.

In addition, the font recognition system utilizes the font tag recognition neural network to generate tag-based font probability vectors. The term “tag-based font probability vector” refers to a set of values that provide a correlation between font tags and known fonts. In particular, the term “tag-based font probability vector” includes an n-dimensional vector where n corresponds to a number of known fonts. For each of the n known fonts, the font probability vector includes a corresponding probability that the text font tag query font matches the known font. In some cases, a font tag recognition neural network generates a tag-based font probability vector as described below.

The term “attention map” as used herein refers to a mapping that connects feature vectors from a first neural network (e.g., the font classification neural network) to a second neural network (e.g., the font tag recognition neural network). In particular, the term “font classification attention map” refers to a font classification feature vector that has been transformed from m-dimensions to n-dimensions. In various embodiments, an implicit font classification attention model (or an implicit font classification attention neural network) receives a font classification feature vector for a font and converts the dimensions of the vector to match the dimensions used by the font tag recognition neural network. In this manner, a font classification attention map enables the font recognition system to mix and weight feature vectors from the font classification neural network with feature vectors of the font tag recognition neural network. Accordingly, the font recognition system can implicitly influence the font tag recognition neural network based on font classification information learned by the font classification neural network for each font.

Referring now to the figures,FIG. 1illustrates the general process for training a font tag recognition neural network and an implicit font classification attention model in accordance with one or more embodiments. In particular,FIG. 1illustrates a series of acts100. In various embodiments, the font recognition system implements the series of acts100. The font recognition system can be implemented by one or more computing devices, such as one or more server devices, one or more client devices, or a combination of server devices and client devices.

As shown inFIG. 1, the font recognition system trains102a font tag recognition neural network to predict font tag probabilities. In various embodiments, the font recognition system utilizes a font training set that includes font images (e.g., input font images), font labels (e.g., font tags and font classifications). For example, the font recognition system utilizes the font tags as a ground-truth to tune the font tag recognition neural network to learn which font images correspond to which font tags. Indeed, the font tag recognition neural network generates font tag probabilities for each input font image that indicates the probability that each font tag correlates to the font displayed in the input font image. Additional detail regarding initially training the font tag recognition neural network is provided below with respect toFIG. 2A.

FIG. 1also illustrates the font recognition system training104a font classification neural network to predict font classifications. In one or more embodiments, the font recognition system utilizes the same font training set mentioned above to train the font classification neural network. For instance, the font recognition system utilizes the font classification labels as a ground-truth to tune the font classification neural network to learn which font images (e.g., input font images) correspond to which font classifications. Indeed, the font recognition system learns to predict a font classification for each input font image. Additional detail regarding training the font classification neural network is provided below with respect toFIG. 2A.

As shown, the font recognition system trains106an implicit font classification attention model to generate a font classification attention map for each font. In some embodiments, the implicit font classification attention model is added to the trained font classification neural network. For example, the font recognition system inputs the predicted font classifications (e.g., font classification coefficients) into the implicit font classification attention model and trains the model to generate a font classification attention map. As mentioned above, a font classification attention map converts or maps the predicted font classifications to match the dimensions of the feature vectors produced by the font tag recognition neural network. In this manner, the font recognition system can incorporate the implicit font information from the font classification neural network to the font tag recognition neural network. Additional detail regarding training the implicit font classification attention model is provided below with respect toFIG. 2B.

To better incorporate the font classification attention map for a font to the font tag recognition neural network, the font recognition system jointly trains108the font tag recognition neural network with the implicit font classification attention model using the font classification attention map to generate enhanced font tag probabilities. More particularly, the font recognition system utilizes joint training to fine-tune the font tag recognition neural network and the implicit font classification attention model to more accurately learn how to incorporate the implicit font information from the font classification neural network into font tag probability predictions. This additional training enables the font recognition system to generate enhanced font tag probabilities that are more accurate and relevant than the initial font tag probabilities obtained from the initially trained font tag recognition neural network. Additional detail regarding joint training is provided below with respect toFIG. 2B.

In various embodiments, the font recognition system can also retrieve one or more fonts in response to a font tag query. For example, the font recognition system obtains a text query that includes one or more font tags. Using the enhanced font tag probabilities, the font recognition system determines fonts in the set of fonts that best correlate to the inputted font tags. The font recognition system can then recommend the one or more retrieved fonts in response to the query. Additional detail regarding retrieving fonts based on font tags is provided below with respect toFIG. 4.

As mentioned above,FIGS. 2A-2Cillustrate diagrams of a more detailed process for training the various font neural networks and models. In particular,FIG. 2Ashows initially training separate font neural networks (e.g., a font tag recognition neural network210and a font classification neural network230) using overlapping training data202.FIG. 2Badds the implicit font classification attention model to the font classification neural network and describes jointly training the font tag recognition neural network with the implicit font classification attention model.FIG. 2Cillustrates an example architecture of the font tag recognition neural network210, which in many embodiments, is similar to the architecture of the font tag recognition neural network.

As mentioned above,FIG. 2Aincludes training data202. As shown, the training data202includes font character images204of fonts as well as font tags206and font classifications208(i.e., font classification labels) that correlate to the fonts. In various embodiments, the font recognition system utilizes portions of the training data202to train the font tag recognition neural network210and the font classification neural network230. For example, in some embodiments, the font recognition system utilizes the font character images204and corresponding font tags206from the training data202to initially train the font tag recognition neural network210. In some embodiments, the font recognition system utilizes the font character images204and corresponding font classifications208to train the font classification neural network230.

In various embodiments, the font recognition system pre-processes the training data202to ensure conformity across the data. In particular, the font recognition system pre-processes font names, tags, and classifications. To illustrate, in one or more embodiments, for each font tag, the font recognition system changes all characters in a tag to lowercase, lemmatizes each word (e.g., removes plurals) in a tag, connects multi-word tags with hyphens (e.g., “sans serif” to “sans-serif”), and/or combines duplicate tags. In additional embodiments, the font recognition system can also filter out tags, such as any tag that appears less than ten times in the training data202.

Before describing how the font recognition system trains each of the font neural networks, additional detail is provided regarding generating or obtaining the training data202. In various embodiments, the font recognition system obtains a set of fonts from one or more sources. For example, the font recognition system obtains fonts from one or more font repositories. As part of obtaining fonts, the font recognition system can also obtain font tags and font classifications corresponding to the fonts. In some instances, the font tags and/or classifications are provided by user input, such as by the creator of the font or by a font curator. In other instances, the tags and/or classifications are automatically generated.

Upon obtaining the set of fonts, the font recognition system can generate the training data202. To illustrate, in one or more embodiments, the font recognition system generates a font character image by selecting a font, renders a number (e.g., five) of random characters (e.g., glyphs including upper and/or lowercase) written in the selected font, and captures/renders an image of the font characters. In addition, the font recognition system can associate one or more font tags and a font classification with the font character image.

In various embodiments, the font recognition system can generate a number of sets of font character images204along with corresponding font tags206and font classifications208for each font in the font set. For instance, the font recognition system renders millions of font character images204paired with font tags206and font classifications208. In additional embodiments, the font recognition system allocates portions of the generated training data for testing (e.g., 10%) and validation (e.g., 10%) in addition to the training portion (e.g., 80%).

In additional embodiments, or in the alternative, the font recognition system can obtain the training data202from a third-party source. For example, the font recognition system previously created a training font set of text images, which is stored in a font database, either locally or remotely. In another example, the font recognition system obtains a training font set from a third-party, such as an online font library or repository. In addition, the font recognition system can combine one or more training font sets with newly rendered text images.

In various embodiments, the font recognition system randomly introduces deviations into the font character images204. For example, the font recognition system randomly introduces noise (e.g., a small Gaussian noise with zero mean and a standard deviation of three), blur (e.g., a random Gaussian blur with standard deviation between two and four), perspective rotation (e.g., a randomly-parameterized affine transformation), and/or shading (e.g., random gradients that fill the input background) into some of the font character images204. In addition, the font recognition system can add variable character spacing and/or variable aspect ratio modifications to the font character images204. These deviations add robustness while training each font neural network as well as enable a trained font tag recognition neural network recognition neural network to better recognize and characters real-world font character images that are later added to be associated with font tags (e.g., offline training).

Turning now to training the font tag recognition neural network210, in various embodiments, the font tag recognition neural network210is a convolutional neural network (CNN). In some embodiments, the font tag recognition neural network is a deep learning convolutional neural network. In alternative embodiments, the font tag recognition neural network210is a different type of neural network.

As shown, the font tag recognition neural network includes lower neural network layers212and higher neural network layers216. In general, the lower neural network layers212are collectively called an encoder and the higher neural network layers216are collectively called a decoder or classifier (e.g., a font tag classifier). In one or more embodiments, the lower neural network layers212are convolutional layers that encode font character images204into hidden encoded features represented as font tag recognition feature vectors214(or simply “tag feature vectors214”).

As just mentioned, the lower neural network layers212generate tag feature vectors214. In various embodiments, the tag feature vectors214are n-dimensional vectors represented in n-dimensional space (i.e., vector space), where n corresponds to the number of font tags included in the collection of font tags206from the training data202. For example, if the training data202included 2,000 font tags, the tag feature vectors214would be a 2,000-dimensional vector. Each dimensional in a tag feature vector provides hidden or latent representations between the font in the font character image and the font tags within a font tag feature/vector space.

The higher neural network layers216can comprise fully-connected layers that classify the tag feature vectors214and output initial font tag probabilities224(e.g., a font tag probability vector). In some embodiments, the higher neural network layers216include a sigmoid function that classifies the probability (e.g., from [0-1]) that each font tag corresponds to a font character image being used to train the font tag recognition neural network210, as indicated in an outputted initial font tag probability of a font tag probability vector. Indeed, the font recognition system utilizes the sigmoid function to classify the font tag probabilities224as the font tag recognition neural network210is designed as a multi-label learning task network to enable multiple tags to be assigned to each font. Overall, the font tag recognition neural network210extracts deep font visual features (e.g., line types, thickness, smoothness, curvature, height, width) and predicts font tag probabilities based on these visual features.

The font recognition system can also employ a loss layers model to train the font tag recognition neural network210. As shown, a font tag recognition loss model222that receives font tags206corresponding to font character images204being fed to the lower neural network layers212. Training the font tag recognition neural network210is now described below.

In one or more embodiments, the font recognition system initially trains the font tag recognition neural network210by tuning parameters based on the font character images204and the font tags206, which are used to generate font tag probabilities. In addition, the font recognition system utilizes the font tag recognition loss model222to provide feedback based on the accuracy of the font classifications, which enables the font recognition system to update the tunable parameters. More specifically, the font recognition system uses error loss feedback to tune font feature extractions in the lower neural network layers212and higher neural network layers216to recognize font tags from input training images.

To demonstrate, the font recognition system trains the lower neural network layers212by tuning font feature extraction parameters, which are used to generate a tag feature vector214for each inputted font character image from the training data202. The font recognition system then provides the generated tag feature vector214to the higher neural network layers216, which compares the generated tag feature vector214to known tag feature vectors learned via training. The higher neural network layers216then utilize font tag recognition parameters to generate an initial font tag probability vector (e.g., initial font tag probabilities), which indicates a correspondence between the input font character image and each font tag.

As mentioned above, during training, the font recognition system provides the initial font tag probability vector to the font tag recognition loss model222. As shown, the font tag recognition loss model222receives font tags206from the training data202, which is utilized as a ground-truth to evaluate the initial font tag probability vector. In particular, the font tag recognition loss model222compares the initial font tag probabilities to the one or more font tags206to determine an amount of font tag recognition error loss (or simply “tag error loss”). In various embodiments, the font tag recognition loss model222employs cross-entropy loss, and/or mean square error computations, or another type of loss formulation to determine the amount of tag error loss.

Also, the font recognition system can utilize the tag error loss to train and optimize the neural network layers of the font tag recognition neural network210via back propagation and end-to-end learning. Indeed, in various embodiments, the font recognition system back propagates the tag error loss to tune tag recognition feature parameters within layers of the font tag recognition neural network210. For instance, in one or more embodiments, the font recognition system takes the tag error loss output from the font tag recognition loss model222and provides it back to the lower neural network layers212and/or the higher neural network layers216until the tag error loss from the font tag recognition loss model222is minimized. In particular, the font tag recognition loss model222provides feedback220to the lower neural network layers212to further tune the font feature extraction parameters and/or the higher neural network layers216to further tune the font tag recognition parameters. As the tag error loss reduces, the accuracy of the initial font tag probabilities224improves.

Equation 1, shown below, provides an example formulation of training the font tag recognition neural network210with cross-entropy loss. By way of context for Equation 1, given the training font set {F1, . . . , FM} and character set {C1, . . . , C52} for an input glyph image Iijthat includes character Ljof font Fi, the font tag recognition neural network210first extracts a hidden feature fijby a CNN. The hidden feature is then fed into a fully-connected layer with N output nodes, where N represents the total tag vocabulary size. Next, a sigmoid unit maps the value of each node in the range of [0-1], which represents the image's probability to match each specific tag. The font recognition system then utilizes cross-entropy loss to train the font tag recognition neural network210as follows:

L=∑i,j⁢∑k=1N⁢(yik⁢log⁡(pij,k)+(1-yik)⁢log⁡(1-pij,k))(1)
As shown in Equation 1, pij,krepresents the predicted probability for Iijto match the kth tags. Also, in Equation 1 above, 1 if Fiis labeled at the kth tag, otherwise yikis 0.

Turning now to the font classification neural network230, the architecture of the font classification neural network230shares many similarities with the font tag recognition neural network210. For example, the font classification neural network230is also a CNN having lower neural network layers232(e.g., convolutional encoder layers) and higher neural network layers236(e.g., a decoder or classifier). However, while the font recognition system trains the font tag recognition neural network210to predict the probability that a given input font (e.g., shown the font character image) corresponds to each of the font tags, the font recognition system trains the font classification neural network230to classify (e.g., identify or predict) the input font shown the font character image. Indeed, the font classification neural network230is designed as a single-label learning task network as each font corresponds to a single font classification (e.g., each font only has one font name).

To achieve the different objectives of the font classification neural network230, the font recognition system employs a few different components and training methods between the two neural networks. To illustrate, in training the font classification neural network230, in one or more embodiments, the font recognition system utilizes the font character images204and corresponding font classifications208from the training data202. As shown, the font recognition system feeds the font character images204into the lower neural network layers232of the font classification neural network230. The lower neural network layers232can utilize font feature extraction parameters to encode font classification feature vectors234.

In various embodiments, the font classification feature vectors234are m-dimensional vectors represented in m-dimensional font classification vector space, where m corresponds to the number of font classifications in the training data202. For example, if the training data202includes 15,000 fonts, the font classification feature vectors234would be a 15,000-dimensional vector. Each dimension in a font classification feature vector provides hidden or latent representations between the font in the font character image and the font classifications within the font classification vector space.

The higher neural network layers236of the font classification neural network230receive the font classification feature vectors234and generate a font classification prediction244. More particularly, in various embodiments, the higher neural network layers236include a softmax classifier that predicts which font classification best corresponds to each given input font (e.g., a font character image). Indeed, the softmax classifier outputs a font classification prediction244(e.g., a font classification prediction vector) that is m-dimensional and all the entries in the vector sum to one (e.g., to accommodate the single-label learning task). The font recognition system selects the entry in the font classification prediction vector with the highest prediction probability as the predicted font for the given input font. In some embodiments, the font recognition system utilizes the cross-entropy loss formulation shown in Equation 1 above with the softmax classifier to train the font classification neural network230.

During training, the font recognition system provides the font classification prediction244to the font classification loss model242. The font classification loss model242compares the identified font indicated in the font classification prediction244to the actual font employed in the training text image to determine an amount of font classification error loss (or simply “classification error loss”). Indeed, the font recognition system can compare the font classification prediction244to font classifications208from the training data202to determine the accuracy and/or classification error loss of the font classification.

In some embodiments, the font classification loss model242utilizes a softmax font classifier cross-entropy loss and/or mean square error computations to determine the amount of classification error loss. For instance, the font classification loss model242identifies when a font classification prediction244is beyond a threshold distance from font features corresponding to the ground-truth font classification (i.e., font classifications208) within the font classification vector space, and/or how far beyond the threshold distance (e.g., classification error loss) the font classification prediction244is from the font classifications208.

Again, using the error loss (e.g., classification error loss) to train and optimize the neural network layers of the font classification neural network230, the font recognition system can utilize back propagation and end-to-end learning to tune feature extraction parameters within layers of the font classification neural network230. For instance, in one or more embodiments, the font recognition system takes the classification error loss output from the font classification loss model242and provides it back to the lower neural network layers232to further tune the font feature extraction parameters and/or the higher neural network layers236to further tune the font classification parameters until the classification error loss from the font classification loss model242is minimized. In this manner, the font recognition system iteratively trains the font classification neural network230to learn a set of best-fit parameters that extract font features from a font character image and accurately classifies (e.g., predicts or identifies) the font within the image.

In one or more embodiments, the font recognition system trains the font tag recognition neural network210and the font classification neural network230separately using the same training data202. In alternative embodiments, the font recognition system concurrently trains the two font neural networks using the same font character images204from the training data202along with other portions of the training data202corresponding to each neural network (e.g., the font tags206with the font tag recognition neural network210and the font classifications208with the font classification neural network230).

In some embodiments, the font recognition system trains one neural network, then adds one or more layers of the second neural network to the first trained neural network and fine-tunes the additional layers. For example, the font recognition system trains or obtains a pre-trained font classification neural network230. Then, the font recognition system adds the neural network layers of the font tag recognition neural network210to the end of the font classification neural network and fine-tunes the overall neural network to generate font tag probabilities.

After training the font tag recognition neural network210and the font classification neural network230(or obtaining a pre-trained font classification neural network), the font recognition system can implicitly inject font classification information from the font classification neural network230into the font tag recognition neural network210to improve the accuracy of the font tag probabilities. To illustrate,FIG. 2Bshows the font recognition system introducing an implicit font classification attention model250that can merge implicit font classification information from the font classification neural network230with the tag feature vectors214of the font tag recognition neural network210.

More particularly,FIG. 2Billustrates a compacted version of the font classification neural network230that represents the font classification neural network230illustrated inFIG. 2A. In some embodiments, the font classification neural network230shown inFIG. 2Bincludes variations from the font classification neural network illustrated inFIG. 2A. For example, the font classification neural network230shown inFIG. 2Bmay include more or less fully-connected layers and/or a different classifier type within the higher neural network layers236.

As mentioned above,FIG. 2Billustrates the implicit font classification attention model250. As shown, the implicit font classification attention model includes higher neural network layers254. In some embodiments, the implicit font classification attention model also includes a classifier within the higher neural network layers254, such as a sigmoid function.

For ease of explanation,FIG. 2Billustrates the implicit font classification attention model being separate from the font classification neural network230. In alternative embodiments, the implicit font classification attention model is embedded into the font classification neural network230, such as added on as a second set of higher neural network layers and/or additional fully-connected layers. For example, in some embodiments, the implicit font classification attention model replaces the softmax classifier of the font classification neural network230and adds one or more additional fully-connected layers as well as a sigmoid classifier (i.e., a sigmoid function). In this manner, the implicit font classification attention model is a sub-network of the font classification neural network230.

In various embodiments, the implicit font classification attention model250converts the m-dimensional output of the font classification predictions244into n-dimensions to match the dimensionality of the font tag recognition neural network210. Indeed, the implicit font classification attention model learns to generate a font classification attention map256that maps m-dimensional font classification predictions vectors into n-dimensions for each font.

In one or more embodiments, the implicit font classification attention model250utilizes a sigmoid function within the higher neural network layers254that outputs a probability vector where each entry ranges from [0-1]. In these embodiments, the implicit font classification attention model250creates an n-dimensional font classification attention map256(based on an m-dimensional input from the font classification neural network230) that provides learned weights that indicates an implicit correspondence between font classifications and font tags for each font.

As mentioned above, the font recognition system can utilize the font classification attention map256to inject implicit font classification information into the font tag recognition neural network210to generate enhanced font tag probabilities. As shown, the font recognition system provides the font classification attention map256for a font to a combiner226within the font tag recognition neural network210. In various embodiments, the combiner226is located prior to the higher neural network layers216of the font tag recognition neural network210(e.g., at the hidden deep feature or node level). In this manner, the font recognition system combines (i.e., at the combiner226) the font classification attention map256and a tag feature vector for the same font within the font tag recognition neural network210before feeding the weighted tag recognition feature vectors228to the higher neural network layers216of the font tag recognition neural network210.

As an example, the font recognition system provides a font character image204of a font to the font tag recognition neural network210and the font classification neural network230. The font tag recognition neural network210generates a tag feature vector214for the font. Concurrently, the font classification neural network230generates a font classification prediction244(or unclassified font classification prediction coefficients), which are provided to the implicit font classification attention model250. The implicit font classification attention model in turn creates the font classification attention map256for the font.

Next, the font recognition system performs element-wise multiplication of the tag feature vector214and the font classification attention map256at the combiner226to obtain a font classification weighted feature vectors228(or simply “weighted feature vector228”). In alternative embodiments, the combiner226combines the tag feature vector214and the font classification attention map256in a manner other than element-wise multiplication. The font recognition system then feeds the weighted feature vector228to the higher neural network layers216(e.g., including a sigmoid function) of the font tag recognition neural network210to generate enhanced font tag probabilities260for the font. In this manner, the weighted feature vector228incorporates the implicit font classification information from the font classification neural network230into the weighted feature vector228, which the font recognition system utilizes to generate the enhanced font tag probabilities260for the font.

In one or more embodiments, the font recognition system jointly trains the font tag recognition neural network210and the implicit font classification attention model250to improve the accuracy of generating enhanced font tag probabilities260. Indeed, the font recognition system jointly trains the font tag recognition neural network210and the implicit font classification attention model250to learn how to optimally extract implicit font classification information (e.g., represented in the font classification attention map256) as well as utilize the implicit font classification information to improve font tag recognition by the font tag recognition neural network210. In this manner, the font recognition system utilizes joint training to fine-tune the font tag recognition neural network210and the implicit font classification attention model250to more accurately learn how to incorporate the implicit font information from the font classification neural network230into font tag probability predictions.

Accordingly, in various embodiments, the font recognition system trains the higher neural network layers216of the font tag recognition neural network210with the implicit font classification attention model250while keeping the lower neural network layers212of the font tag recognition neural network210and the font classification neural network230fixed. For example, during training, the font recognition system provides tag error loss from the font tag recognition loss model to the higher neural network layers216and the implicit font classification attention model250. In some embodiments, the font recognition system alternates providing the tag error loss between the higher neural network layers216and the implicit font classification attention model250until one or both networks converge. In alternative embodiments, the font recognition system jointly trains by providing the tag error loss simultaneously to the higher neural network layers216and the implicit font classification attention model250.

As mentioned above, in jointly training the higher neural network layers216of the font tag recognition neural network210with the implicit font classification attention model250, the font recognition system fixes the tunable parameters of the lower neural network layers212of the font tag recognition neural network210and the font classification neural network230. In some embodiments, the font recognition system further tunes one or more of these layers/networks. For example, the font recognition system trains the font classification neural network230in connection with training the layers of the implicit font classification attention model250.

By way of additional context with respect to Equation 1, for each glyph Iij, the font recognition system indicates the predicted font class distribution, which is represented as cij(e.g., a font classification feature vector234). Next, the font recognition system feeds cijinto the implicit font classification attention model250. As mentioned above, in some embodiments, the implicit font classification attention model250includes a fully-connected layer with a sigmoid unit, which transforms cijinto a font classification attention map256, represented as Bij, where Bijhas n-dimensions matching the hidden feature vector fijdescribed above from the font tag recognition neural network210. Further, each of the n-elements or nodes within Bijhas a range of [0-1].

FIG. 2Bprovides one example architecture of neural networks for the font recognition system to generate enhanced font tag probabilities260. The font recognition system can alternatively employ other architectures. For example, in one embodiment, the font recognition system concurrently trains the font tag recognition neural network210and the font classification neural network230using a shared error loss model in a multi-task framework to capture the font class knowledge (i.e., implicit font classification information) within the font tag recognition neural network210. In another embodiment, the font recognition system first trains the font classification neural network230, then fine-tunes the neural network for tag probability predictions (e.g., enhanced font tag probabilities260). Other architectures and configurations are also possible.

As shown inFIG. 2C, an example architecture of the font tag recognition neural network210is illustrated. As mentioned above, in many embodiments, this architecture is similar to the architecture of the font classification neural network230. To provide context,FIG. 2Cincludes the font recognition system providing the training data202to the font tag recognition neural network210that generates the initial font tag probabilities224, as described above.

In particular, the font tag recognition neural network210inFIG. 2Cshows that lower neural network layers212includes five convolutional layers. In some embodiments, a rectified linear unit (ReLU) non-linearity is applied to the output of each convolutional and fully connected layer. In addition, in various embodiments, the font tag recognition neural network210includes two normalization layers and/or two max-pooling layers.FIG. 2Calso includes example neuron dimensions for each layer (i.e., 48×48×64 neurons for the first convolutional layer).

The font recognition system feeds the tag feature vectors214outputted from the lower neural network layers212to the higher neural network layers216, as explained earlier. As shown, the higher neural network layers216include fully-connected layers (i.e., fc6, fc7, fc8) and a classifier function258. As shown, the first two fully-connected layers are 4,096-dimensional while the last fully-connected layer is 2,383-dimension. In this example, 2,383 indicates 2,383 font tags (or 2,383 fonts in the case of a font classification neural network230). In this manner, the classifier function258outputs a 2,383-dimension initial font tag probability vector.

As mentioned above, the architecture between the font tag recognition neural network210and the font classification neural network230can be similar. For example, in some embodiments, the architecture is the same between the two neural networks except for the classifier function258. For example, in the case of the font tag recognition neural network210, the classifier function258is a sigmoid function. In the case of the font classification neural network230, the classifier function258is a softmax classifier.

Moreover, in some embodiments, the font recognition system utilizes the ResNet-50 architecture as the basic CNN architecture for the font tag recognition neural network210and/or the font classification neural network230. In alternative embodiments, the font recognition system utilizes the ResNet-18 architecture. Further, in some embodiments, the font recognition system can employ a learning rate of 0.00005 for the convolutional layers and of 0.0005 for the fully-connected layers.

FIGS. 2A-2Cdescribed various embodiments of jointly training the font tag recognition neural network with an implicit font classification attention model. Accordingly, the actions and algorithms described in connection withFIGS. 2A-2Cprovide an example structure and architecture for performing a step for jointly training the font tag recognition neural network210with an implicit font classification attention model250within the font classification neural network230to generate a font classification attention map256for each of the plurality of fonts that extracts and applies implicit font information from the training font images to the font tag recognition neural network210.

Indeed,FIGS. 2A-2Cprovide detail for identifying training font images (e.g., font character images204) of a plurality of fonts, where each training font image includes a font tag (e.g., font tags206) and a font classification (e.g., font classifications208) of a font included in the training font image. Further,FIGS. 2A-2Cprovide detail for training a font tag recognition neural network210to determine initial font tag probabilities224for each font of the plurality of fonts using the font training images and corresponding font tags as well as training a font classification neural network230to classify the plurality of fonts using the font training images and corresponding font classifications. Furthermore,FIGS. 2A-2Cprovide detail for generating, by the trained font tag recognition neural network, enhanced font tag probabilities260for a font of the plurality of fonts based on the font classification attention map256.

Turning now toFIG. 3, additional detail is provided regarding creating a more robust font classification attention map for each font included in the training data202. More particularly, the actions described in connection withFIG. 3primarily relate to training the font classification neural network230and the implicit font classification attention model250, as described above in connection withFIG. 2B.

As described previously, the font recognition system provides a font character image to the font classification neural network230(e.g., indicated by the bold arrow). In additional embodiments, the font recognition system generates additional font character images having the same font as the provided font character image. To illustrate,FIG. 3shows the font recognition system first providing the font character image to a font character image generator360. For example, the font recognition system provides the font character image from the font character images204of the training data202to the font character image generator360. In another example, the font recognition system provides an indication of the font included in the font character image to the font character image generator360.

The font character image generator360can generate additional font character images that utilize the same font included in the original font character image. In one or more embodiments, the font character image generator randomly selects multiple strings of characters (e.g., at least 5 random glyphs per string) from the font and renders the strings as additional font character images. For example, the character image generator360generates four additional images per font. The font recognition system then provides the additional font character images to the font classification neural network230and the implicit font classification attention model250(e.g., indicated by the non-bold arrows) to generate font classification attention maps (e.g., font classification attention map256) for each of the additional font character images.

Further, the font recognition system can combine the font classification attention map for the font character image with the font classification attention maps for the additional font character images to obtain an averaged font classification attention map356. For example, in one or more embodiments, the font recognition system averages the font classification attention maps to determine the averaged font classification attention map356, which is then provided to the font tag recognition neural network210at the node level, as previously described above in connection withFIG. 2B. The averaged font classification attention map356can better extract the implicit font classification information for the font than a single font classification attention map. In this manner, the averaged font classification attention map356provides robustness that enables the font tag recognition neural network210to generate more accurate enhanced font tag probability vectors.

ExplainingFIG. 3in another way, for the input image Iij, the font recognition system randomly selects another x images Iij1, . . . , Iijx(e.g., 4 additional images) with different characters Lj1, . . . , Ljxbut all with the same font (i.e., Fi). The font recognition system then predicts font classification attention maps for each image, represented as Bij1, . . . , Bijx. Further, the font recognition system then calculates the averaged font classification attention map356using the formulation shown in Equation 2 below. In Equation 2, ⊙ represents an element-wise multiplication between each font classification attention map.
Bi=Bij⊙Bij1⊙ . . . ⊙Bijx(2)

Then, using the averaged font classification attention map356(i.e., Bi), the font recognition system injects the averaged font classification attention map356into the font tag recognition neural network210. In particular, the font recognition system can perform an element-wise multiplication between fijand Bito obtain the re-weighted hidden feature of Iij(e.g., a weighted tag recognition feature vector228)), which the font recognition system then feeds into the top fully-connected layer of the font tag recognition neural network210, as described above. Overall, the random selection of multiple glyph images of a single font improves the accuracy and discriminability of the averaged font classification attention map356for the font.

Upon jointly training the font tag recognition neural network210with the injected implicit font classification information, the font recognition system can utilize the enhanced font tag probabilities for each font to retrieve fonts based on text queries of font tags. As mentioned above, conventional systems detect or predict fonts based on an input image. In contrast, the font recognition system can retrieve one or more fonts based on a text query rather than an image query. To illustrate,FIG. 4shows a diagram of retrieving fonts based on a font tag query in accordance with one or more embodiments.

As shown,FIG. 4includes a font tag query402(i.e., the font tag of “Decorative”), a trained font tag database404, and a recommended font406(i.e., the font of “Kapelka”). In various embodiments, the font recognition system receives the font tag query402as a text query from a user (or another entity). In response, the font recognition system utilizes the trained font tag database404to identify one or more recommended fonts that correspond to the font tag query402. Further, the font recognition system provides the identified one or more recommended fonts (e.g., the recommended font406) to the user in response to the text query.

The trained font tag database404includes a collection of enhanced font tag probabilities for each font and font tag in the training data. As described above, the font recognition system trains the font tag recognition neural network to generate enhanced font tag probabilities for each font, where the enhanced font tag probabilities (from 0%-100%) for a font indicate the probability that the font corresponds to each font tag. In various embodiments, the font recognition system stores the correlation between each font and font tag in the trained font tag database404. The font recognition system repeats these actions for each font in the training data with respect to all the font tags.

The trained font tag database404can include various tables and/or be organized in a variety of configurations. For example, the font recognition system includes a first table that lists each font along with the probability that each font tag corresponds to a given font. Further, in additional embodiments, the font recognition system generates a second table (e.g., converts the first table) that includes a listing of the font tags, where each font tag indicates the probability that each font corresponds to the font tag. In this manner, the font recognition system can quickly identify a font tag from a font tag query402in the trained font tag database404and retrieve the font having the highest probability correspondence with the identified font tag. Similarly, the font recognition system can retrieve and recommend the top x number of fonts that correspond to the identified font tag (e.g., the font tag from a font tag query).

In some embodiments, the font tag query402includes multiple font tags. In these embodiments, the font recognition system can utilize the trained font tag database404to retrieve and recommend the font that best corresponds to the multiple font tags. For example, in one or more embodiments, the font recognition system identifies each of the multiple font tags in the font tag query402and extracts the corresponding font probabilities for the multiple font tags. In addition, the font recognition system can sum, average, or otherwise normalize the corresponding font probabilities between the multiple font tags to identify one or more highest (e.g., most favorable) fonts that best suit the multiple font tags in the font tag query402.

When the font tag query402includes multiple font tags, the font recognition system can apply different weights to each of the font tags in the font tag query402. To illustrate, in one or more embodiments, the font recognition system weights the font tags based on commonality, usage, and/or popularity. For example, the font recognition system weights more common font tags over less common font tags. Alternatively, the font recognition system weights less common font tags over the more common font tags. In this manner, the font recognition system can give greater weight to some font tags in the font tag query402when retrieving and recommended fonts in response to a font tag query402.

The font recognition system can occasionally update the trained font tag database404to include new fonts, font tags, or to improve existing learned font tag retrieval data. For example, the font recognition system identifies a new font missing from the trained font tag database404. The font recognition system can generate font character images and ground-truth data as new training data and feed the new training data to the trained font tag recognition neural network, trained font classification neural network, and the trained implicit font classification attention model. Using the actions and processes described above in connection withFIG. 2B, the font recognition system can generate enhanced font tag probabilities for the new font. In addition, the font recognition system can update the trained font tag database404to include the new or updated enhanced font tag probabilities.

In various embodiments, the font recognition system can provide and utilize the trained font tag database404on a mobile client device to retrieve fonts. Indeed, the font recognition system can utilize the trained font tag database404to retrieve and recommend fonts to a user in response to font tag queries without the need to create, train, execute, or store the font tag recognition neural network and/or font classification neural network, which some modern mobile client devices would struggle to accomplish. In this manner, the font recognition system can train and generate the trained font tag database404on a server device and provide the compact trained font tag database404to a mobile client device, enabling most modern mobile client devices to quickly and efficiently retrieve fonts in response to font tag queries.

Referring now toFIG. 5, additional detail will be provided regarding capabilities and components of the font recognition system (i.e., tag-based font recognition system) in accordance with one or more embodiments. In particular,FIG. 5shows a schematic diagram of an example architecture of the tag-based font recognition system504(or simply “font recognition system504”) located within a font management system502and hosted on a computing device500. The font recognition system504can represent one or more embodiments of the font recognition system described previously.

As shown, the font recognition system504is located on a computing device500within a font management system502. In general, the computing device500may represent various types of client devices. For example, in some embodiments, the client is a mobile device, such as a mobile telephone, a smartphone, a PDA, a tablet, a laptop, etc. In other embodiments, the computing device500is a non-mobile device, such as a desktop or server, or another type of client device. In some embodiments, portions of the computing device500correspond to computing devices of different types (e.g., some components operate on the computing device500when acting as a server device and some components operate on the computing device500when acting as a client device). Additional details with regard to the computing device500are discussed below as well as with respect toFIG. 10.

The font management system502, in general, facilitates the creation, modification, sharing, installation, receipt, and/or deletion of digital fonts within electronic documents and/or system applications. For example, the font management system502stores a repository of fonts on the computing device500, such as in a font database (not shown). In addition, the font management system502can access additional fonts located remotely. Further, in some embodiments, the font management system502can be located separately from the computing device500and provide fonts to the computing device500. In one or more embodiments, the font management system502comprises ADOBE® TYPEKIT®.

In addition, the font management system502can operate in connection with one or more applications to display fonts on the computing device500. For example, in one or more embodiments, the font management system502provides fonts to a word processing application such as ADOBE® ACROBAT®, ADOBE® INDESIGN®, or another word processing application. In other embodiments, the font management system502provides fonts to a design application such as ADOBE® ILLUSTRATOR®.

As illustrated inFIG. 5, the font recognition system504includes various components. For example, the font recognition system504includes a font manager506, a font tag recognition neural network508, a font classification neural network510, an implicit font classification attention model512, a font tag recommender514, and a storage manager516. Each of these components is described below in turn.

The font manager506can store, receive, detect, install, order, and/or organize fonts within the computing device500. For example, in one or more embodiments, the font manager506stores a set of fonts on the computing device500. In some embodiments, the font manager506, in connection with the font management system502, maintains fonts within a font database. For example, the font manager506maintains a set of fonts that a user can employ in an electronic document. In an additional example, the font manager506maintains a training font set518. In various embodiments, the font manager506can identify and access additional fonts not stored or located on the computing device500.

The font manager506can generate the training font set518used to train the font tag recognition neural network508, the font classification neural network510, and the implicit font classification attention model512. For example, the font manager506renders font character images520from random characters (i.e., glyphs) for each font in a font set, as previously described. In addition, the font manager506associates font tags522and font classifications524with each of the rendered font character images520. In some embodiments, the font manager506stores the generated training font set518(including font character images520, font tags522, and font classifications524) in the storage manager516.

As shown inFIG. 5, the font recognition system504includes the font tag recognition neural network508. As described above, the font recognition system trains the font tag recognition neural network508to learn a correlation between fonts and font tags. The font tag recognition neural network508can include multiple neural network layers, such as convolutional layers, fully-connected layers with a sigmoid function classifier, and loss layers. Example architecture of the font tag recognition neural network508is provided above with respect toFIGS. 2A-2C.

In addition, the font tag recognition neural network508can generate feature vectors526, such as font tag recognition feature vectors that encode deep or hidden visual effects of fonts in relation to font tags. In addition, the font tag recognition neural network508can generate font tag probability vectors530. For example, the font tag recognition neural network508first generates initial font tag probability vectors, then upon further training generates enhanced font tag probability vectors that incorporate implicit font classification information (e.g., from a font classification attention map528, as described above.

As also shown inFIG. 5, the font recognition system504includes the font classification neural network510. As described above, the font recognition system trains the font classification neural network510to learn to classify fonts based on font images. The font classification neural network510can include multiple neural network layers, such as convolutional layers, fully-connected layers with a softmax function classifier, and loss layers. Example architecture of the font tag recognition neural network508is provided above with respect toFIGS. 2A-2C. The font classification neural network510can also generate feature vectors526(e.g., font classification feature vectors) and font classification prediction vectors, as described previously.

In addition,FIG. 5shows the implicit font classification attention model512. As explained earlier, the implicit font classification attention model512converts implicit font classification information from the font classification neural network510into a font classification attention map528, which is injected into the node level of the font tag recognition neural network508via the font classification attention map528. The font classification neural network510can include fully-connected layers with a sigmoid function classifier. Example architecture of the implicit font classification attention model is provided above with respect toFIG. 2B.

As shown, the font recognition system504includes the font tag recommender514. The font tag recommender514can receive fonts in response to a font tag query. For example, the font tag recommender514identifies one or more font tags within a trained font tag database532, determines corresponding fonts, and retrieves the fonts best matching the font tag query. In addition, the font recognition system504provides the retrieved fonts as recommended fonts in response to the font tag query.

Further, as shown, the font recognition system504includes the storage manager516. The storage manager516communicates with other components of the font recognition system504to store, maintain, and access data used to train the font neural networks and models disclosed herein (e.g., the training font set518, feature vectors526, the font classification attention map528, and the font tag probability vectors530). In addition, the storage manager516stores and maintains the trained font tag database532, which is described above.

Each of the components506-532of the font recognition system504can include software, hardware, or both. For example, the components506-532can include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices, such as a client device or server device. When executed by the one or more processors, the computer-executable instructions of the font recognition system504can cause the computing device(s) to perform the feature learning methods described herein. Alternatively, the components506-532can include hardware, such as a special-purpose processing device to perform a certain function or group of functions. Alternatively, the components506-532of the font recognition system504can include a combination of computer-executable instructions and hardware.

Furthermore, the components506-532of the font recognition system504may, for example, be implemented as one or more operating systems, as one or more stand-alone applications, as one or more modules of an application, as one or more plug-ins, as one or more library functions or functions that may be called by other applications, and/or as a cloud computing model. Thus, the components506-532may be implemented as a stand-alone application, such as a desktop or mobile application. Furthermore, the components506-532may be implemented as one or more web-based applications hosted on a remote server. The components506-532may also be implemented in a suite of mobile device applications or “apps.” To illustrate, the components506-532may be implemented in an application, including but not limited to ADOBE® TYPEKIT®, ADOBE® INDESIGN®, ADOBE ACROBAT®, ADOBE® ILLUSTRATOR®, ADOBE PHOTOSHOP®, ADOBE® CREATIVE CLOUD® software. “ADOBE,” “INDESIGN” “ACROBAT,” “ILLUSTRATOR,” “PHOTOSHOP,” and “CREATIVE CLOUD” are either registered trademarks or trademarks of Adobe Inc. in the United States and/or other countries.

FIG. 6illustrates a schematic diagram of an environment600in which the tag-based font recognition system504(or simply “font recognition system504”) may be implemented in accordance with one or more embodiments. In one or more embodiments, the environment600includes various computing devices including server device(s)602and one or more client devices604a,604b. In addition, the environment600includes a network606. The network606may be any suitable network over which the computing devices can communicate. Example networks are discussed in more detail below with regard toFIG. 10.

As illustrated inFIG. 6, the environment600includes the server device(s)602, which may comprise any computing device, such as one or more of the computing devices described below in relation toFIG. 10. In addition, the server device(s)602includes the font management system502and the font recognition system504, which are described previously. For example, as described above, the font recognition system504can train and apply font tag recognition neural networks to learn correlations between fonts and font tags based on implicit font classification information as well as retrieve fonts based on a font tag query.

In addition, the environment600includes the one or more client devices604a,604b. The client devices604a,604bmay comprise any computing device, such as the computing device described below in relation toFIG. 10. As described above, the one or more client devices604a,604bcan utilize the trained font tag database to retrieve fonts based on a font tag query.

As illustrated, in one or more embodiments, the server device(s)602can include all, or a portion of, the font recognition system504. In particular, the font recognition system504can comprise an application running on the server device(s)602or a portion of a software application that can be downloaded from the server device(s)602. For example, the font recognition system504can include a web hosting application that allows a client device604ato interact with content hosted on the server device(s)602. To illustrate, in one or more embodiments of the environment600, the client device604aaccesses a web page supported by the server device(s)602. In particular, the client device604acan run an application to allow a user to access, view, select, identify, and/or recommend a font from a font tag query within a web page or website hosted at the server device(s)602(e.g., a web page enables a user to provide a font tag query, and in response recommends one or more fonts).

AlthoughFIG. 6illustrates a particular arrangement of the server device(s)602, the client devices604a,604band the network606, various additional arrangements are possible. For example, whileFIG. 6illustrates the one or more client devices604a,604bcommunicating with the server device(s)602via the network606, in one or more embodiments a single client device may communicate directly with the server device(s)602, bypassing the network606.

Similarly, although the environment600ofFIG. 6is depicted as having various components, the environment600may have additional or alternative components. For example, the font recognition system504can be implemented on multiple computing devices. In particular, the font recognition system504may be implemented in whole by the server device(s)602or the font recognition system504may be implemented in whole by the client device604a. Alternatively, the font recognition system504may be implemented across multiple devices or components (e.g., utilizing the server device(s)602and the one or more client devices604a,604b).

Turning now to the next figure,FIG. 7. illustrates sample retrieved fonts700based on font tag queries in accordance with one or more embodiments. In particular,FIG. 7includes font tag queries702and recommended fonts704retrieved by the font recognition system504in response to the font tag queries702, as described above. Indeed, the recommended fonts704include the top-20 fonts that correspond to each of the font tag queries702.

Indeed,FIG. 7shows qualitative results of the font recognition system504, where the font recognition system504performance well with respect to font retrieval across a variety of font tag queries702(e.g., user font tag inputs). For example, results of recommended fonts704from category-related tags are shown for the tags “sans-serif,” “script,” and “handwritten.” In addition, results of recommended fonts704from utility-related tags are shown for the tag “headline.” Further, results of recommended fonts704from characteristic-related tags are shown for the tags “block” and “computer.”

With respect to quantitative results, as mentioned above, researchers compared embodiments of the font recognition system to baseline neural networks and found that the font recognition system outperformed these baseline neural networks. More particularly, the researchers evaluated the effectiveness of the font recognition system504based on a standard retrieval measure of mean average precision (MAP). To illustrate, using the MAP measure, the researchers evaluated M tag queries for a specific query q. By way of context, if the positive fonts f1f2, f3. . . fNreceive probability ranks of r1, r2, r3. . . rN, then, the average precision score of q is computed as shown in Equation 3 below.

Average⁢⁢Precisionq=∑n=1N⁢nrn(3)
The font recognition system504then calculates the mean values of each the average precision score for each query to obtain the MAP.

To evaluate the embodiments of the font recognition system to the baseline neural networks, the researchers obtained a training set of 19,161 fonts and 1,923 font tags. In particular, the researchers split the 19,161 fonts from the dataset into a training set (80%), a validation set (10%), and a test set (10%), where each font was only included in one of the sets. The researchers then utilized two font tag query lists to evaluate the performance of the different models. The first font tag query list included all 1,923 tags, where each tag was used as a font tag query (i.e., MAP). For each font tag query, the font was marked as positive if the ground-truth tag list contained the corresponding tag of the font tag query. The second font tag query list was limited to frequently used font tags, which have a higher tendency to be searched by users. Specifically, the second font tag query list included the 300 most frequently selected tags in the training set (e.g., MAP-300).

For each of the font tag query lists, the font recognition system504computed the MAP using Equation 3, as described above. Table 1 below includes the MAP score comparisons between embodiments of the font recognition system to the baseline neural networks. As shown in Table 1, the baseline neural networks include a baseline pre-trained font tag recognition neural network that does not include any information from a font classification neural network (e.g., trained to generate initial font tag probabilities), a multi-tasked font classification neural network concurrently trained with a font classification neural network, and a pre-trained font classification neural network fine-tuned to perform font tag recognition. Each of these baseline models is described above in connection with embodiments of the font recognition system504. In addition, Table 1 includes the font tag recognition neural network jointly trained with implicit font classification attention model (shown in bold), as described above, that generates enhanced font tag probabilities.

As shown in Table 1, compared with other baseline font tag recognition neural network models, the font tag recognition neural network jointly trained with the implicit font classification attention model outperformed the baseline models with respect to both font tag query lists (e.g., MAP and MAP-300). Indeed, the jointly trained font tag recognition neural network of the font recognition system504disclosed herein achieved remarkably better performance than the baseline model, the multi-task model, and the fine-tuned model, which demonstrates its effectiveness to accurately retrieve fonts.

Further, while not shown, the researchers found that the jointly trained font tag recognition neural network of the font recognition system504disclosed herein achieved better results over the baseline models with respect to normalized discounted cumulative gain (nDCG). In particular, the jointly trained font tag recognition neural network of the font recognition system504disclosed herein particularly performed better with respect to single-tag font tag queries.

FIGS. 1-7, the corresponding text, and the examples provide several different systems, methods, techniques, components, and/or devices of the font recognition system504in accordance with one or more embodiments. In addition to the above description, one or more embodiments can also be described in terms of flowcharts including acts for accomplishing a particular result. For example,FIG. 8andFIG. 9illustrate flowcharts of an example sequence of acts in accordance with one or more embodiments. In addition,FIG. 8andFIG. 9may be performed with more or fewer acts. Further, the acts may be performed in differing orders. Additionally, the acts described herein may be repeated or performed in parallel with one another or parallel with different instances of the same or similar acts.

WhileFIG. 8andFIG. 9illustrate a series of acts according to particular embodiments, alternative embodiments may omit, add to, reorder, and/or modify any of the acts shown. The series of acts ofFIG. 8andFIG. 9can be performed as part of a method. Alternatively, a non-transitory computer-readable medium can comprise instructions, when executed by one or more processors, cause a computing device (e.g., a client device and/or a server device) to perform the series of acts ofFIG. 8andFIG. 9. In still further embodiments, a system performs the series of acts ofFIG. 8andFIG. 9.

In addition, the series of acts ofFIG. 8andFIG. 9can be implemented on one or more computing devices, such as the computing device500or the server device(s)602. In addition, in some embodiments, the series of acts ofFIG. 8andFIG. 9can be implemented in a digital environment for creating or editing electronic documents. In various embodiments, the series of acts ofFIG. 8andFIG. 9are implemented on a computing device having a memory that stores digital fonts (e.g., font training images corresponding to a plurality of fonts as well as font tags and font classifications corresponding to the font training images). In some embodiments, the memory stores a font classification neural network trained to classify the plurality of fonts using the font training images and the corresponding font classifications.

To illustrate,FIG. 8shows a flowchart of a series of acts800for training a tag-based font recognition system to generate an enhanced font tag probability in accordance with one or more embodiments. As shown, the series of acts800includes an act810of training a font tag recognition neural network to determine initial tag-based font probabilities for each font. In particular, the act810can involve training a font tag recognition neural network to determine initial tag-based font probabilities for each font of the plurality of fonts using the font training images and corresponding font tags. In various embodiments, the font tag recognition neural network includes lower neural network layers and higher fully-connected neural network layers.

In one or more embodiments, the dimensions of font tag recognition feature vectors generated by the lower neural network layers of the font tag recognition neural network include n-dimensions based on a first number of font tags corresponding to the font training images. In various embodiments, the font tag recognition neural network utilizes a first sigmoid probability function to generate the font tag recognition feature vectors. In some embodiments, the font tag recognition neural network includes a first convolutional neural network.

As shown, the series of acts800also includes an act820of training the font classification neural network to classify the fonts. In particular, the act820can involve training the font classification neural network to classify the plurality of fonts using the font training images and corresponding font classifications. In one or more embodiments, the dimensions of the font classification prediction vectors generated by the font classification neural network include m-dimensions based on a second number of font classifications corresponding to the font training images, where m-dimensions is different from n-dimensions. In some embodiments, the font classification neural network utilizes a softmax probability function to generate the font classification prediction vectors. In various embodiments, the font classification neural network includes a second convolutional neural network.

As shown inFIG. 8, the series of acts800further includes an act830of training an implicit font classification attention model to obtain an attention map that transforms font classification prediction vectors. In one or more embodiments, the act830includes training an implicit font classification attention model using font classification prediction vectors outputted from the trained font classification neural network to obtain a font classification attention map that transforms the font classification prediction vectors to match dimensions of font tag recognition feature vectors outputted from the trained font tag recognition neural network. In various embodiments, the implicit font classification attention model includes additional fully-connected layers and a sigmoid probability function added to the font classification neural network.

In many embodiments, the implicit font classification attention model utilizes a second sigmoid probability function to generate the font classification attention map. In various embodiments, the act830includes determining the font classification attention map for a first font of a first training image of the training font images. In additional embodiments, determining the font classification attention map for the first font includes generating a second training image including random characters of the first font, determining a second font classification attention map for the first font, averaging the first font classification attention map and the second font classification attention map for the first font, and utilizing the averaged font classification attention map for the first font to jointly train the font tag recognition neural network with the implicit font classification attention model.

As shown, the series of acts800also includes an act840of jointly training the font tag recognition neural network with the font classification attention map to determine enhanced tag-based font probabilities for each font. In particular, the act840can involve jointly training the font tag recognition neural network with the implicit font classification attention model using the font classification attention map and the font classification prediction vectors to determine enhanced tag-based font probabilities for each font of the plurality of fonts. In some embodiments, the act840includes comparing the font tags corresponding to the font training images as a ground-truth to the enhanced tag probabilities determined for the plurality of fonts to obtain an error loss amount and back propagating the error loss amount to the higher fully-connected neural network layers of the font tag recognition neural network.

In some embodiments, the act840includes jointly training the higher fully-connected neural network layers of the font tag recognition neural network with the implicit font classification attention model. In various embodiments, the act840also includes applying element-wise multiplication between the font classification attention map and the font classification prediction vectors to obtain weighted font classification feature vectors. In additional embodiments, the weighted font classification feature vectors are utilized to train the higher fully-connected neural network layers of the font tag recognition neural network. Additionally, in some embodiments, the font classification feature vectors and the weighted font classification feature vectors are hidden feature vectors.

The series of acts800can also include a number of additional acts. In one or more embodiments, the series of acts800includes the acts of receiving a font tag query including a font tag; determining, based on the enhanced tag-based font probabilities for each font of the plurality of fonts, one or more fonts having a highest tag-based font probability corresponding to the font tag in the font tag query; and providing the one or more fonts in response to the received font tag query.

As mentioned above,FIG. 9illustrates a flowchart of a series of acts900for recommending fonts based on a font tag query in accordance with one or more embodiments. As shown, the series of acts900includes a first set of acts910of generating enhanced tag-based font probability vectors and a second set of acts920of recommending fonts. The first set of acts910can be performed offline or prior to the second set of acts920, which are performed online or in response to receiving a font tag query.

As shown the first set of acts910comprises an act930of extracting font tag recognition feature vectors from the input font images. In particular, act930can involve extracting font tag recognition feature vectors from input font images using an encoder of a font tag recognition neural network. Each input font image can comprise characters in a font of a plurality of fonts. Act930can involve extracting font tag recognition feature vectors having a first dimensionality. For example, the font tag recognition feature vectors can comprise n-dimensions based on a first number of font tags used to train the font tag recognition neural network.

The first set of acts910comprises an act932of generating font classification prediction vectors. More specifically, act932involves generating font classification prediction vectors by processing the input font images using a font classification neural network. Act932involve generating font classification prediction vectors having a second dimensionality, which differs from the first dimensionality. For example, the font classification prediction vectors can comprise m-dimensions based on a number of font classifications used to train the font classification neural network. Act932can involve generating the font classification prediction vectors utilizing a SoftMax probability function.

The first set of acts910also comprises an act934of generating font classification attention maps. In particular, act934can involve generating font classification attention maps by transforming the font classification prediction vectors using an implicit font classification attention model. For example, act934can involve generating the font classification attention maps by transforming the font classification prediction vectors from the second dimensionality to the first dimensionality. In various embodiments, the act934includes generating a robust or averaged font classification attention maps by combining multiple font classification attention maps of the font generated from different character strings of the font.

The first set of acts910comprises an act936of generating font classification weighted feature vectors. In particular, act936involves generating font classification weighted feature vectors by combining the font tag recognition feature vectors and the font classification attention maps. For example, act936can involve applying element-wise multiplication between the font classification attention maps and the font tag recognition feature vectors.

The first set of acts910comprises an act938of determining enhanced tag-based font probability vectors. In particular, act938involves determining enhanced tag-based font probability vectors by processing the font classification weighted feature vectors utilizing the higher neural network layers of the font tag recognition neural network. For example, act938can involve generating the enhanced tag-based font probability vectors utilizing a sigmoid probability function. The first set of acts910can further involve generating a font tag database, based on the enhanced tag-based font probability vectors, that indicates the probability that each tag is associated with each font.

The second set of acts920includes an act940of receiving a font tag query. For example, act940can involve receiving one or more tags (e.g., words/phrases that a user desires in a font) entered by a user at a client device. Act940can optionally involve receiving the font tag query at a server over a network. Alternatively, act940can comprise receiving the font tag query at a client device.

The second set of acts920also involves an act942of determining one or more fonts having high probabilities of being associated with a font tag from the font tag query. In particular, act942involves determining one or more fonts having high probabilities of being associated with a font tag from the font tag query based on the enhanced tag-based font probability vectors. For example, act942can involve accessing a font tag database that was been generated prior to receiving the font tag query. The font tag database can be based on the enhanced tag-based font probability vectors and indicate the probability that each tag is associated with each font.

Finally, the second set of acts920can involve an act944of providing the one or more fonts as a recommended fonts. For example, act944can involve returning the one or more fonts as a response to the font tag query.

The term “digital environment,” as used herein, generally refers to an environment implemented, for example, as a stand-alone application (e.g., a personal computer or mobile application running on a computing device), as an element of an application, as a plug-in for an application, as a library function or functions, as a computing device, and/or as a cloud computing system. A digital medium environment allows the font recognition system to train and employ multiple neural networks and/or machine-learning models, as described herein.

A cloud computing model can be composed of various characteristics such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, and so forth. A cloud computing model can also expose various service models, such as, for example, Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). A cloud computing model can also be deployed using different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, and so forth. In addition, as used herein, the term “cloud computing environment” refers to an environment in which cloud computing is employed.

FIG. 10illustrates a block diagram of an example computing device1000that may be configured to perform one or more of the processes described above. One will appreciate that one or more computing devices, such as the computing device1000may represent the computing devices described above (e.g., computing device500, server device(s)602, and client devices604a-b). In one or more embodiments, the computing device1000may be a mobile device (e.g., a mobile telephone, a smartphone, a PDA, a tablet, a laptop, a camera, a tracker, a watch, a wearable device, etc.). In some embodiments, the computing device1000may be a non-mobile device (e.g., a desktop computer or another type of client device). Further, the computing device1000may be a server device that includes cloud-based processing and storage capabilities.

As shown inFIG. 10, the computing device1000can include one or more processor(s)1002, memory1004, a storage device1006, input/output (“I/O”) interfaces1008, and a communication interface1010, which may be communicatively coupled by way of a communication infrastructure (e.g., bus1012). While the computing device1000is shown inFIG. 10, the components illustrated inFIG. 10are not intended to be limiting. Additional or alternative components may be used in other embodiments. Furthermore, in certain embodiments, the computing device1000includes fewer components than those shown inFIG. 10. Components of the computing device1000shown inFIG. 10will now be described in additional detail.

In particular embodiments, the processor(s)1002includes hardware for executing instructions, such as those making up a computer program. As an example, and not by way of limitation, to execute instructions, the processor(s)1002may retrieve (or fetch) the instructions from an internal register, an internal cache, memory1004, or a storage device1006and decode and execute them.

The computing device1000includes memory1004, which is coupled to the processor(s)1002. The memory1004may be used for storing data, metadata, and programs for execution by the processor(s). The memory1004may include one or more of volatile and non-volatile memories, such as Random-Access Memory (“RAM”), Read-Only Memory (“ROM”), a solid-state disk (“SSD”), Flash, Phase Change Memory (“PCM”), or other types of data storage. The memory1004may be internal or distributed memory.

The computing device1000includes a storage device1006includes storage for storing data or instructions. As an example, and not by way of limitation, the storage device1006can include a non-transitory storage medium described above. The storage device1006may include a hard disk drive (HDD), flash memory, a Universal Serial Bus (USB) drive or a combination these or other storage devices.

As shown, the computing device1000includes one or more I/O interfaces1008, which are provided to allow a user to provide input to (such as user strokes), receive output from, and otherwise transfer data to and from the computing device1000. These I/O interfaces1008may include a mouse, keypad or a keyboard, a touch screen, camera, optical scanner, network interface, modem, other known I/O devices or a combination of the I/O interfaces1008. The touch screen may be activated with a stylus or a finger.

The computing device1000can further include a communication interface1010. The communication interface1010can include hardware, software, or both. The communication interface1010provides one or more interfaces for communication (such as, for example, packet-based communication) between the computing device and one or more other computing devices or one or more networks. As an example, and not by way of limitation, communication interface1010may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI. The computing device1000can further include a bus1012. The bus1012can include hardware, software, or both that connects components of computing device1000to each other.