Customer Segmentation Data Science | by Arun Jagota | Towards Data Science

In marketing, customer segmentation is the process of grouping customers by common traits. Discerning buying habits by customer type helps to market appropriately. For instance, it reveals the sizes of the various segments, how much we make from them, etc. This can help decide how to apportion the marketing budget.
In data science, clustering is the process of grouping objects by some common traits.
See the connection? Clustering — a data science method — is a good fit for customer segmentation — a use case. That said, as often happens, there is more to it when we start digging. We call this modeling.
Let’s start by looking at specific traits we’d want to segment on. We’ll distinguish between B2B marketing and B2C marketing as the traits vary.
B2B Marketing
The customers are businesses, i.e. companies. One set of traits found useful to segment by are the so-called firmographics: industry, location, company size (small, medium, large, etc).
B2C Marketing
Here, the customers are consumers, i.e. people. Traits useful to consider for segmentation are the so-called demographic ones: age, gender, location, ethnicity, income level, and so on.
Modeling
Now, we abstract away from the specifics of B2B vs B2C and look at the modeling per se. While we will use B2B examples for illustration, the modeling is more general.
Single Structured Factor
First consider a single trait, say industry. We wish to segment customers just by this trait. First, let’s assume industry has a small number of clean values. (“Clean” means from a controlled vocabulary — such as ones in a drop-down list.)
This segmentation is easy to do. It just involves grouping and aggregating.
Single Unstructured Factor
Now suppose the industry values are not clean. They are stored in a text field in free-form. At data entry, whatever value makes sense is put.
Say there are 100K customer records in our database, and most have industry field values filled in.
Immediately our segmentation problem has become way harder. The same industry will often get expressed in different ways in different records. Word order may vary; acronyms may get used; connectives may or may not be present; synonyms may get used.
Strings Similarity Measure: A first reasonable attempt at solving this problem goes something like this. First, we define a similarity measure on pairs of industry strings. We start by defining an industry string’s set-of-words. This is just the set of distinct tokens in the string. People familiar with the term bag-of-words might think of this as the bag with word frequencies removed. Next, given two word-sets, we need a sensible way to quantify their similarity. Here is one: It is called Jaccard similarity.
The Jaccard Similarity of two sets is the number of elements they have in common divided by the number of elements in their union.
Below is an example.
Jaccard-coefficient({predictive analytics platform},{predictive analytics}) = 2⁄3
Grouping Records into Clusters: Next, we need to group the records into clusters using this similarity measure. Pairs of records in the same cluster should be similar to each other; pairs in different clusters not.
This is easier said than done. It’s not clear that we have even fully thought through our problem formulation yet. I’ll elaborate. The word ‘group’ seems to imply that a record should go into exactly one cluster. Is this really what we want? Not necessarily. Imagine an industry value Medical Informatics. We might want to place this in multiple clusters — some medical-related, some IT-related.
Industries at Multiple Levels: The values in the industry field may span multiple levels of an industry taxonomy. For example, one record may set the industry field value to a broad category, say Software. Another record may specify the industry at a finer-level, say Database Software. So maybe we want to do hierarchical clustering, not flat clustering. So a record whose industry is Database Software may be assigned to a cluster that is a descendant of the cluster whose industry is Software.
Clustering Algorithms: Which algorithm to use depends on which type of clustering we want — flat or hierarchical? Even within the flat category, the algorithm depends on whether we want the clustering to be hard or soft. (Hard meaning an item belongs to exactly one cluster.)
Flat Clustering: Algorithms suited to hard clustering include k-means clustering, dbscan, connected components clustering, and others. Algorithms for soft clustering are often softened versions of the hard clustering ones. One such is soft — also called probabilistic — k-means clustering. This is based on a powerful algorithm call the EM algorithm of which (hard) k-means clustering is a special case.
Hierarchical Clustering: Popular hierarchical clustering algorithms include agglomerative (bottom-up) clustering or divisive (top-down) clustering. In the former, items are first clustered into tight clusters, then clusters are clustered into coarser clusters. This process is repeated until we have a tree of clusters. Its leaves are the items. The tree’s root is a single cluster representing the entire data set.
Divisive clustering works in the opposite direction. It starts from the cluster representing the entire data set; a cluster is then split into two. The resulting clusters become the children of the former. Thus the tree of clusters is grown top-down.
Bottom-up vs Top-down: One might think, bottom-up or top-down, they are mirror image processes, so why to care which to use. The choice does matter. It is generally easier to merge two suitable small clusters than to split a large cluster into two good ones. This fact alone largely explains why bottom-up clustering is far more popular than top-down clustering.
Another important consideration is that bottom-up vs top-down clustering can produce different results. Building up bottom-up is not the same as splitting top-down. For this reason, top-down clustering is sometimes considered when bottom-up’s results are not sufficiently good, or we just want to know how much switching from bottom-up to top-down changes the tree in any particular instance.
Linguistic Structure In Industry Names: Coming back to our industry with unstructured values example, we might take a third tack. Take advantage of the linguistic structure implicitly present in industry names. This will probably work better. We illustrate this below on a toy example. The algorithm that emerges may be easily enhanced to accommodate more realistic scenarios.
Example: Imagine there are just six distinct industry values: Software, Database Software, Analytics Software, Manufacturing, Auto Manufacturing, Drug Manufacturing. (Some of these names are contrived, and not to be taken seriously.) The sought hierarchical structure is
Software ← Database Software, Software ← Analytics SoftwareManufacturing ← Auto Manufacturing, Manufacturing ← Drug Manufacturing
where p ← c denotes a parent-child instance.
Algorithm Leveraging Linguistic Structure: Below is the algorithm. It will yield our desired hierarchy in this example.
Order the set of distinct industry values by the number of words in them. Least number of words first. Denote this ordering as I1, I2, ..., Ik.For each i in 1..k   If Ii does not have a parent, set Ii as a root of the hierarchy.   For each j in (i+1)..k      If Ij’s parent is not set and is_child(Ij,Ii)          parent(Ij) = Ii
Let’s illustrate this algorithm in our example. First, we sort the industry names by the number of words in them. This yields the ordered list
Software, Manufacturing, Database Software, Analytics Software, Auto Manufacturing, Drug Manufacturing
Next, we make Software a root and scan to the right of it to find all its children. We discover
Software ← Database Software, Software ← Analytics Software
Next, we make Manufacturing a root of this hierarchy and scan to its right to find all its children, skipping over industry names that have already been parented. We discover
Manufacturing ← Auto Manufacturing, Manufacturing ← Drug Manufacturing
Next, we try to find the children of each of the two-word industry names. We don’t find any. So we are done.
Multiple Factors, Mixed Structure
Now consider segmenting companies by the combination of multiple factors. Specifically, by the key firmographics: industry, location, and company size. Some of these factors might be structured, others unstructured. For example industry, as before, is unstructured. Company size may be structured: represented by a fixed number of values such as small, medium, large. This complicates the problem significantly.
Multi-factor Similarity Metric: First, let’s define a suitable metric that quantifies how similar a pair of factor tuples is. This definition needs to be thought-through with care, as this metric will hugely influence which companies get grouped into which segments.
Preamble 1 — Factor Descriptions: First, let’s refine our factor descriptions. We’ll keep industry unstructured as before. We’ll make company size ordinal. This just means company size will have a fixed set of ordered values. As an example, small, medium, large. We can later add more values to company size, such as very small and very large. We can even have the set of values be automatically determined via a data-driven company size binning process. The resulting values might differ from data set to data set. So long as the values can be ordered, we are good. (By the way not every set of values can be ordered. Consider gender = female, male. There is no natural order. We could impose some artificial order but that might not go so well.)
For location, let’s keep it simple, to begin with, and just choose 5-digit zip. (This does limit our companies to being in the USA only.)
Preamble 2 — Some Notes on the Metric: Next, we define a suitable metric under these factor definitions. It’s important to note that should we modify our factor definitions, the metric might require large-scale revision. As an example, say we define location as a combination of zip, city, state, country. Quantifying how similar two locations are is now much more involved.
Similarity Metrics for Individual Factors
We will take the approach of defining similarity metrics for each factor individually, and upon these build our multi-factor similarity metric.
Common Part — Sigmoid Function: Before we dive into the individual metrics, we’d like to mention a key point that cuts across their design. In all of them, we use a sigmoid function as a soft approximation to a step function. We do this because we want the similarity to rapidly drop towards 0 when the values being compared are insufficiently similar. This gives us control over the similarity detection versus false-positive rates.
Because of this, let’s first define the sigmoid function.
sigmoid_{a,b}(x) = 1/(1+exp(-a*(x-b)))
a controls the sigmoid’s steepness, i.e. the extent to which it approximates the step function. bcontrols the soft threshold. Just as a step function, the sigmoid discriminates values flanking b.
Company Names Similarity Metric: Let’s start with the metric for comparing two company sizes. Let’s rank order the company size values as 1, 2, 3, ..., k. That is, 1 denotes the smallest company size bin and k the largest company size bin. Our similarity metric is
S(i,j) = sigmoid_{a,b}(-|i-j|)
Here a>0and b>0 give us control over the sigmoid’s steepness and the threshold. We have used -|i-j|rather than |i-j|as the argument, because we seek an inverted sigmoid behavior. We could instead absorb this negative sign into the a, but making it explicit is more intuitive.
S(i,j) has the simple intuition that the closer the bins i and j are in their rank order, i.e. in the sizes of the companies they represent, the higher the similarity value.
We’ll choose sensible defaults so you can forget about them if you don’t like free parameters. Specifically, we’ll set a to 1, the most commonly used sigmoid steepness. To better reveal what value of bmakes sense let us rewrite the sigmoid as sigmoid(b-|i-j|). We will set bto 1.5. This will get us the behavior that when |i-j|<=1.5the sigmoid’s value will cross 0.5 from below. We can, of course, fine-tune this behavior as we wish.
Zips Similarity Metric: Next, let’s work on a metric for zip values. We’ll take advantage of the fact that 5-digit zips are numbers and, generally speaking, the closer the numbers are the closer the zip locations are. We use
S(z1,z2) =sigmoid_{a,b}(-|z1-z2|)
This time we will choose a>0to be much less than 1 and b to ~ 20. The intuition is that we want S(z1,z2) to be greater than 0.5 even when |z1-z2| is around 20, roughly speaking. Plus we want the crossing to have low steepness. We can, of course, fine-tune these values as we see fit.
Substitution and Transposition Errors: While this metric models proximity well enough, it does not account for substitution or transposition errors. Let’s start by seeing an example of each. (We have intentionally omitted insertion and deletion errors because they change the number of digits.)
Substitution error: 21034, 21834, Transposition error: 21034, 20134
As in the examples above, we’ll consider only single-digit substitution errors and adjacent-digit transposition errors. Here is a simple metric that accommodates single digit substitution errors:
S2(z1,z2) = sigmoid_{a,b}(-sum_i NE(z1[i],z2[i]))
Here NE(x,y) equals 0 when xequals yand 1 when not. We will set ato 1 and bto 1.5, as the behavior we seek (on the sigmoid’s argument) is exactly what we sought for company size similarity.
Similarly, we can define a metric that accommodates up to one adjacent-digit transposition.
T(z1,z2) = sigmoid_{a,b}(-#adjacent-digit-transpositions(z1,z2))
We use the same aand bas we did for substitution scoring.
So now we can combine all these metrics into a single metric.
Sim(z1,z2) = Max(S(z1,z2),S2(z1,z2),T(z1,z2) )
Industries Similarity Metric: Finally, for industry, we will use the Jaccard coefficient on their word-sets as the similarity metric, further transformed via a sigmoid. That is
S_industry(I1,I2) = sigmoid_{a,b}(Jaccard-coefficient(I1,I2))
We will not describe how to choose the default values of a and bin detail. Rather, we will just note that ashould be positive and greater than 1 and b should be chosen to be the value of the Jaccard-coefficient we deem to be at the edge of similarity. Note that the Jaccard-coefficient returns a value between 0 and 1 where 0 denotes maximally dissimilar and 1 denotes maximally similar.
Multi-factor Metric
We are now ready to build our multi-factor metric on top of these metrics. First, let’s briefly discuss the pros and cons of this approach. This modular design makes it easy to add new factors or to delete existing ones. It also makes it easy to adjust the relative influence of the similarities of individual factor values towards the overall metric. That said, it does not naturally model interactions among factors. To model these, one will need to explicitly add in specific interaction terms.
Our multi-factor metric is
S(C1,C2) = w_zip*S_zip(C1.zip,C2.zip) + w_industry*S_industry(C1.industry,C2.industry) + w_company_size*S(C1.company_size_bin_rank,C2.company_size_bin_rank)
The factor weights let us control the relative influences of the individual factor similarities in the overall metric.
Specializations: Now that we have this metric, we can build various specializations of it by simply setting certain factor weights to 0. So we can realize all single-factor versions — similarity-by-zip, similarity-by-industry, similarity-by-company-size — and also all two-factor versions. Armed with these specializations we can build segments more flexibly. Effectively we can group by one factor, by two factors, or by all three factors. Note that“group by” really means “fuzzy group by”.
Clustering the Matrix of Similarities: The key point we’d like to bring out is that the natural way to leverage the pairwise similarity metrics we have defined is to compute pairwise similarities among all pairs of companies and capture these in a similarity matrix. (That is unless the number of companies is so large that computing pairwise similarities will run too slow. We will assume this is not the case.)
Clustering algorithms may then work directly on this matrix. This separates concerns. Clustering algorithms only operate on a similarity matrix, not caring how it was produced. The similarity metrics focus on quantifying the actual similarities, not caring how objects will be clustered downstream.
A Specific Algorithm — Graph Partitioning: Let’s see a concrete clustering algorithm on a similarity matrix. We’ll express the matrix as a graph. The graph’s nodes are the objects, in our case companies. Two nodes are connected by an edge if the corresponding objects are sufficiently similar. The weight on an edge captures the actual similarity value.
The high-level idea of the algorithm is to partition the graph into dense subgraphs connected by sparse crossing regions. Dense subgraphs will represent tight clusters. Sparse crossing regions will ensure that these clusters are well-separated.
Let’s refine this description. We need some key concepts first.
Subgraph Density: We define the density of a subgraph on k nodes as the sum of the weights on the edges that connect pairs of nodes within this set divided by k(k-1)/2. This assumes that the weight on an edge is between 0 and 1. Non-edges may be viewed as edges with weight 0. Under this representation, the density of a subgraph is the average weight of an edge in it.
Node Weights: We define the weight of a node as the sum of the weights of the edges touching the node. We define the weight of a node on a node-set as the sum of the weights of edges touching this node whose other endpoint is in that set. As an example, the weight of node a on nodes b, c, and d is the sum of the weights of the edges a-b, a-c, and a-d.
The Actual Algorithm: Next, we describe a simple heuristic algorithm that finds a good clustering.
find_one_good_cluster(G)   C = {v}, where v is a highest-weight node in G   Repeat     Find a node v in G-C whose weight on C is     sufficiently high.     If no such v exists        Close C.       remaining_clusters = find_one_good_cluster(G-C)       Return remaining_clusters + {C}     Else       Add v to C   forever
The “sufficiently high” test biases the algorithm to find tight clusters. (Tightness is not guaranteed as the tightness check is done only when a node is added to a cluster.) The algorithm does not explicitly favor cross-cluster sparsity. This happens implicitly. A cluster keeps growing as long as a node outside of it has sufficiently high cumulative weight to nodes in it. Consequently, when a cluster stops growing, all nodes outside it have low cumulative weight to nodes in it. This favors sparse cross-cluster regions.
Summary
In this post, we have seen how clustering as a data science method is a good fit for market segmentation as a use case. We’ve looked into factors to use for B2B vs B2C segmentation. We have examined issues that arise when a factor has clean values; when a factor has a long tail of unstructured values; and when a factor has a small set of ordinal values.
We have discussed flat clustering, soft clustering, and hierarchical clustering. We have looked at how domain knowledge can guide us into what algorithms to use, even to the point of moving us away from standard algorithms. (I am referring to clustering industry names into a hierarchy of industries leveraging the linguistic structure implicit in these names.)
We have presented specific metrics for quantifying the similarities of some types of factors we encounter in B2B settings. We have then presented a multi-factor similarity measure, and described how to use it in multi-factor clustering (think segmenting a market by combinations of multiple factors).