Characteristic Based Similarity for New Products Forecasting | by Rafael Cordeiro | Towards Data Science

“How much I’m gonna sell of it?” is the question that every retailer has in mind when they are thinking about adding a new material into their stores and e-commerce.
We usually have a lot of techniques to help us with forecasting and prevision, but when we talk about new products we face a little big problem that is the fact if we don’t have sales data for the new product to use in regression or in a time series algorithm.
So... how can we forecast it?
One way to work around this lack of data is finding a product that is similar to the new one and copying a percentage of its historical sales data, and with it, you can now apply your preferred forecasting technique.
Basically, you can compare the products in several different ways and besides that each business will have to define what are the ideal features to use in this analysis.
After having the features defined, which is the most difficult and important step of the process, you will follow a simple step-by-step to achieve your goal:
1. Apply one-hot encoding to categorical data
Applying one hot encoding we go from comparing distances between the categories names/texts, to comparing whether or not the product is in the same category as the old one.
2. Apply a scaler technique to numerical data
When we talk about distances it’s always something like A-B=C, but now we are discussing about the distance of several characteristics and the result sounds like the sum of all Cs created by all As minus all Bs.
There would be no problem with that, only if the range of values from columns is all the same, and I’m quite sure that this is very, very rare to occur.
Let’s say that we have 3 columns, one has a range of values from 200 to 400, another column is something like 15k to 78k, and a third one is from 1 to 10.
This is going to create a problem due to the fact of the difference between ranges is going to result in different scales of distance, in this case, 100s, 10000s, and 10s respectively, and if we try to apply a distance metric without a scaler applied the second column would have a degree of importance bigger than the other columns.
So, don’t forget to apply it.
3. Calculate the distances between the new product and the old ones
I think that this is the part that you were expecting the most, right?
Well, if it is I expect to not disappoint you, because the solution is really simple.
Again, the first thing you have to do here is to define what kind of distance metric you want to use. See a list of simple and useful distances below:
Euclidian
Manhattan
Minkowski
You must choose which one is the best distance metric for you before continuing.
For this project, I used the Euclidean distance so we end with the whole thing in just two lines of code:
idx_closest = np.argmin(    [np.linalg.norm(new_obs_arr - np.array(x)) for x in df_old.values])df_aux = df_old.iloc[idx_closest]\    .to_frame()\    .transpose()\    .reset_index()
idx_closest is the index value of the reference material of the old data frame, and with it, we get the entire corresponding row and stores it at df_aux.
4. Reverse one-hot encoding
How the process considers for which plant code the material is registered we have to revert the process of one hot encoding at least for the plant code column.
To revert it we should:
Define the columns created by the one hot encoding process.
Select only the columns that have ‘1’ in it.
Convert its name to the original one.
Delete the columns created by the one hot encoding at the beginning of the process.
def reverse_one_hot_encoder(df, col_nm):    """    Reverse the one hot encoding process        Parameters    ----------    df : pandas.DataFrame        data frame object with the old products that are more similar the new products    col_nm : str        name of the original column            Returns    -------        df_result : pandas.DataFrame        data frame object with the old products that are more similar the new products without the OHE procress    """    lst_stack = [x for x in df.columns if x.find(col_nm)>=0]    df_aux = df[lst_stack]    df[col_nm] = df_aux[df_aux==1]\        .stack()\        .reset_index()\        .drop(0,1)['level_1']\        .apply(lambda x: int(x.replace(col_nm+'_','')))\        .to_list()    for col in lst_stack:        df = df.drop(col, axis=1)    return df
5. Calculate the Scaling Factor
And now we have achieved the Scaling Factor step, also known as the sleep taker due to the number of hours of sleep that I’ve lost thinking about it.
Ok... Ok... I’m not being fair with this guy, but I really had lost some time thinking about it, to at the end, result in something that looks too easy to do, so it might just be my internal anger trying to release itself a little bit here.
Here is why I’ve lost so much time on it...
Well, remembering the A-B=C from the Distance step we could guess that “Ok, now I have the new row and the old row defined. So why don’t I, to define the scaling factor, just divide one per other and get a percentage result of it?”
The mindset is fine, but the application of it not so much because instead of comparing two numbers now we are trying to get the percentage from two arrays that are entire data frames rows and there is no trivial way to do this.
So the approach we’ve chosen to defeat this problem was to define a math function which decreases as the distance, that we already have calculated, increases.
We thought that penalize more the great distances than the low ones should be a good thing, so we tried “e^(-distance2)” and if you are an attentive person you’ve noticed that I said “tried” and that means it doesn’t have worked well.
The problem was that we thought that every reference material chosen as similar should really be close to the new one, right? But in some cases, we do not have so many materials to compare and we end up having the best what we can have and not a perfect match.
So, to not be so aggressive with the distance growth we’ve added the average distance value of the resulting group to get a smother decrease ratio.
# import Libsimport matplotlib.pyplot as pltimport statisticsimport numpy as np# simulate distance increasingx = np.array(np.arange(0, 100, 0.001))# get average value from distancesavg = statistics.mean(x)# plot linesplt.plot(x, np.exp(-(x**2)), 'b-')plt.plot(x, np.exp(-(x**2/avg**2)), 'r-')plt.legend(labels=['Original', 'Smothed'])# add labelsplt.xlabel("Distance")plt.ylabel("Saling Factor")plt.show()
Another important point is to look at the product price because if the new product’s price is lower than the reference this means that it has a chance of selling more, but if it is bigger the chance is of selling less.
To aggregate this chance of selling more or less to the scaling factor we’ve decided to calculate how much the price of the old product represents of the new one.
Once we have the scaling factor based on the math function and the price ratio we multiply one per other to get a final scaling factor.
The algorithm exports a .csv with five columns:
location
product
reference_location_code
reference_code
scaling_factor
You can use it to simulate historical sales data to your new product and then apply any forecasting algorithm on it!
Schedule a meeting to talk with your master data team or whoever is the team that registers/creates new products in your database, because they could easily tell you what are the initial characteristic of new materials.
Choose wisely the materials that could be a reference to a new product. The ones that you have stopped to work with, but still existing in your database or products that are new too (less than two or three months since the first sale) may not be the best decision because they probably don’t have a good historical sales curve.
Choose a product hierarchy to filter this way you speed up the cooperation time once you reduce the comparison sample and probably will compare products that are more similar instead of searching in your whole product base.
Don’t forget that once your new product starts selling it is going to receive real sales data and you should start using it to forecast instead of the one you generated.
Well, with this I think you can now a bit better your new products by finding a similar old material to it and scaling its sales to use as an input to your forecast tool!
I hope this article has helped you!
And you can see the code of this project here:
github.com
Special thanks to Pierre Palud who put up with me while I cleared a million of questions about the best way to define the scaling factor