A simple Introduction to Multilayer Perceptron and Autoencoder for Estimating Used Car Prices with Deep Learning for Beginners
Source: tai_blog

How can we estimate the price of objects such as used cars as accurately as possible?   In addition to traditional methods based on statistical and heuristic approaches (e.g. comparison method  cost approach or expert evaluation)  machine learning and deep learning models offer new alternatives. Such models can process large amounts of data efficiently and recognize complex patterns in the data  some of which are difficult for us humans to identify. Another important advantage of these models is that they can be continuously updated with the latest data. In my previous article Machine Learning Models to Predict Used Car Prices explained: A Beginners Guide  I already presented the most common machine learning models such as Linear Regression  Decision Tree  Random Forest  Gradient Boosting Machines  XGBoost and Support Vector Regression.   In this article  I will give you a simple 10-minute introduction to the most important deep learning models that are frequently used in recent research (see reference) to predict the prices of used cars.   The task for the various models is to estimate the price of used cars (second-hand cars) as accurately as possible based on the available data. Possible characteristics are brand  model of the car  year of manufacture  mileage  engine power  fuel type  etc. This task is a regression problem  as the value to be estimated  the price of the car  is continuous.   Deep Learning Models for the Prediction of PricesIn the latest research (see reference)  deep learning models such as Multilayer Perceptron (MLP) and Autoencoder are used for the price estimation of used cars.   Multilayer Perceptron (MLP)This model is an artificial neural network consisting of several layers of neurons. Each layer consists of neurons  nodes  that are connected to each other by weighted connections. MLPs are feedforward neural networks. In these networks  information only flows in one direction.   How a Multilayer Perceptron model worksThe input layer takes in the data: In our example  where we want to predict the price of used cars  these are the features such as brand  model  year of manufacture  mileage  engine power  fuel type  etc. This input data is passed through the neurons of the input layer to the neurons of the first hidden layer.The hidden layer lies between the input layer and the output layer. The model can consist of one or more hidden layers  each responsible for learning complex patterns in the data. To achieve this  non-linear transformations are performed within them. Each neuron in the hidden layers calculates a weighted sum of the inputs  adds a bias and applies an activation function. The bias is an additional parameter that helps the model to generalize better and allows the neuron to adjust its activation threshold. A typical activation function is ReLU. The activation function determines whether a neuron is activated and then performs the nonlinear transformation of the inputs. The transformed values are passed through subsequent layers until they reach the output layer.The output layer returns the prediction: In our example  this layer returns the predicted price for the specific vehicle.PROS of MLPsNonlinearity: By using nonlinear activation functions  MLPs can recognize complex  nonlinear patterns in the data.Scalability: With MLPs  you can add more layers and neurons to scale the model to learn complex patterns. This makes the model suitable for large and diverse data sets.CONS of MLPsResources: Especially if the model contains many hidden layers and neurons  training can be computationally intensive.Data requirements: Using MPLs with small datasets (e.g. less than a few thousand rows) can lead to overfitting. Especially for more complex tasks  you need a large amount of training data so that your model can learn effectively.Hyperparameter tuning: Especially for beginners  optimizing hyperparameters can be time-consuming and complex.Autoencoder-ModelAn autoencoder is also a neural network. This model is mainly used for unsupervised learning. An autoencoder uses an encoder to capture the most significant features of the input data and compress them into a simplified representation. The decoder then attempts to reconstruct the original data from this compressed representation.   How an Autoencoder model worksThe input layer takes in the data: In our example  where we predict the price of used cars  the input data consists of features such as brand  model  year of manufacture  mileage  engine power  fuel type  etc. This input data is forwarded to the neurons of the first hidden layer.The encoder compresses the input data: The input data is compressed into a low-dimensional representation (also called latent space or bottleneck). Unless you want to become a professor of autoencoders  you do not need to understand in detail what is happening here  especially as a beginner. The encoder consists of several layers that gradually reduce the dimension of the data. Each of these layers performs non-linear transformations by calculating a weighted sum of the inputs  adding a bias and applying an activation function.Data in compressed form in the latent space: The latent space represents the compressed form of the input data. This compressed representation should capture the most important features of the input data so that the decoder can reconstruct the original data with the highest possible accuracy.Reconstructing the data in the decoder: The decoder takes the compressed data from the latent space and attempts to reconstruct its original form. The decoder is like a mirror image of the encoder and performs similar transformations  but in reverse order  to restore the data.The output layer provides the prediction: In the example of used car price estimation  the output layer returns the prediction of the prices based on the compressed representation of the features.PROS of AutoencoderFeature detection: Car encoders are very good at capturing the most important features of the data and removing irrelevant information. This can be useful in the used car price prediction task to identify the most important influencing factors.Dimension reduction: Large data sets can be processed more efficiently and model performance can be improved as autoencoders can be used as a dimension reduction technique.CONS of AutoencoderComputational intensity: Training autoencoders can be very computationally intensive  especially for large and complex datasets.Difficulty in reconstruction: If the input data is highly variable  it can be difficult for an autoencoder model to accurately reconstruct this data. The model may not be able to capture all the details of the input data  which can lead to inaccurate predictions. For example  in our used car price prediction example  the data set could consist of cars from many different brands  models  years of manufacture and different mileages and engine outputs. This large variety in the dataset can make it difficult for the autoencoder to learn an accurate latent representation that takes all these differences into account. If the model cannot accurately reconstruct the input data  this means that important information is lost. And this in turn can affect the accuracy of the price prediction.Overfitting: Especially with small data sets  there is a risk of overfitting (as with all neural networks).Tips for implementing a multilayer perceptron model or an autoencoderIf you are a newbie and want to try your hand at these models  I have put together some tips to help you implement a multilayer perceptron model or autoencoder.   Conduct an exploratory data analysis (EDA) Start with an EDA to better understand your data. Analyze the distribution of the features in your dataset  and check for missing values and outliers.   Clean up missing values Remove missing values or replace them with estimates. For example  you can replace missing values with the mean or median of the corresponding column or  in the case of time series data  with the preceding or subsequent value.   Normalize numerical characteristics Bring numerical features to a comparable scale that the model can learn efficiently. This is important because the numerical features often have different units and orders of magnitude. There are two common scaling methods for this: min-max scaling or z-scaling (standardization).   Min-max scaling  Scaling of the data to a range from 0 to 1:   # Min-Max Scaling for normalization of features from sklearn.preprocessing import MinMaxScaler scaler = MinMaxScaler() df[['Kilometerstand'  'Motorleistung']] = scaler.fit_transform(df[['Kilometerstand'  'Motorleistung']])Z-scaling (standardization)  Centering the data around the mean value and scaling with the standard deviation:   # Z-Scaling of features from sklearn.preprocessing import StandardScaler scaler = StandardScaler() df[['Kilometerstand'  'Motorleistung']] = scaler.fit_transform(df[['Kilometerstand'  'Motorleistung']])Bring categorical features into a numerical form When estimating the price of used cars  you consider categorical characteristics such as car brands or fuel types.   With the one-hot encoding method  each category is converted into a binary column  For example  a category fuel type with the values gasoline  diesel and electric  is then converted into three columns  where each column is 0 or 1:   Gasoline -> [1 0 0]  Diesel -> [0 1 0]  Electric -> [0 0 1]   With the label encoding method  each category is converted into a unique numerical value  For example  gasoline is converted to 0  diesel to 1 and electric to 2:   Gasoline -> 0  Diesel -> 1  Electric -> 2   Determine the activation function The activation function determines whether a neuron is activated and performs non-linear transformations to allow the model to learn complex patterns in the data. One of the most commonly used activation functions is ReLU. In our example (used car price estimation)  you do not need to use an activation function (e.g. Sigmoid  Tanh) in the output layer because the model must be able to output a wide range of continuous values to estimate the price of the used car.   Use tools to avoid overfitting Overfitting is if your model is only well fitted to the training data but performs poorly on new data. For example  you can add L2 regularization and a dropout in the hidden layers:   With an L2 regularization  you force the model to learn simpler patterns. This regularization adds a penalty for large weight values.If you add a dropout  a certain number of neurons will be randomly deactivated during training. This is to prevent the model from relying too heavily on certain neurons.Early stopping stops the training as soon as the performance is no longer improved.Integrate batch normalization To stabilize and accelerate the training  you can integrate batch normalization after each shift.   # Example of an activation function activation_function = nn.ReLU() # Tool to avoid overfitting (weight decay is a regularization technique) linear_layer = nn.Linear(in_features  out_features  bias=True) linear_layer.weight_decay = 1e-5 # Example of Dropout (another regularization technique to avoid overfitting) dropout_layer = nn.Dropout(p=0.5) # Example of Batch Normalization (used to normalize the input of each mini-batch to improve training stability and speed) batch_norm_layer = nn.BatchNorm1d(num_features)Specific for Multilayer PerceptronWith MLPs  many hyperparameters can be customized. Start with a simple model architecture and only gradually increase the complexity. With MLPs  you can set several hyperparameters to optimize the performance of the model. Check out the most important hyperparameters in the image:   It is best to start with a simple configuration and systematically optimize the hyperparameters. For example  you could start with a learning rate of 0.001  a ReLu activation function and 2 hidden layers  each containing 128 neurons per layer.   Specific for AutoencoderYou can also customize many hyperparameters for autoencoder models. Therefore  start with a simple model architecture and gradually increase the complexity. See the most important hyperparameters below:   For example  start with this configuration: Set a learning rate of 0.001  a ReLu activation function  2 layers in the encoder and decoder with 128 neurons per layer and a bottleneck layer with 32 neurons.   Optimize the latent representation A central part of the autoencoder is the latent space or bottleneck. This is the compressed representation of the input data and is often much smaller than the original data. The purpose of this compression is to capture the most important features of the data and remove irrelevant information. This step is important because a well-optimized latent representation allows the autoencoder to accurately reconstruct the data and learn relevant patterns. For example  if we have 20 features of used cars in our example  we can reduce the latent space to 5 neurons to compress the most important information of these 20 features.   If the size is too small  important information may be lost.If the size is defined too large  it can lead to overfitting.Define the encoder and the decoder The encoder takes the input data and compresses it into the latent representation. The decoder takes the compressed data and attempts to return the data to its original form. The decoder often has a similar but reversed structure to the encoder.   # Example for Encoder import torch.nn as nn class Encoder(nn.Module): def __init__(self): super(Encoder  self).__init__() self.encoder = nn.Sequential( nn.Linear(20  64)  # 20 Eingabemerkmale auf 64 Neuronen reduzieren nn.ReLU()  nn.Linear(64  32)  # Weiter auf 32 Neuronen reduzieren nn.ReLU()  nn.Linear(32  5) # Schlielich auf 5 Neuronen (latenter Raum) reduzieren ) def forward(self  x): return self.encoder(x) # Example for Decoder class Decoder(nn.Module): def __init__(self): super(Decoder  self).__init__() self.decoder = nn.Sequential( nn.Linear(5  32)  # Vom latenten Raum (5 Neuronen) zu 32 Neuronen nn.ReLU()  nn.Linear(32  64)  # Weiter zu 64 Neuronen nn.ReLU()  nn.Linear(64  20) # Schlielich zurck zu den 20 Ausgangsmerkmalen ) def forward(self  x): return self.decoder(x)Calculate the reconstruction loss The reconstruction loss measures how well the autoencoder can reconstruct the input data after compression and decompression. It is calculated by measuring the difference between the original data and the reconstructed data.   Low reconstruction loss: The autoencoder can reconstruct the data well.High reconstruction loss: The autoencoder cannot reconstruct the data well.Where is the best place to continue learning?Multilayer Perceptron  Datacamp Tutorial (free)Multilayer Perceptron  YouTube TutorialAutoencoder  Datacamp Tutorial (free)ConclusionMultilayer Perceptron and Autoencoder are both neural networks that have recently been used to estimate the prices of used cars. Im curious whether one of the machine learning models will achieve better results or one of these two deep learning models. Do you have experience with either of these models?   References   Study: Using Artificial Neural Network  A Deep Learning Approach for Used Car Price PredictionStudy: Using Artificial Neural Network  Prediction Of Used Car Prices Using Artificial Neural Networks And Machine LearningStudy: Using Multilayer Perceptron  A Multimodel Transfer-Learning-Based Car Price Prediction Model with an Automatic Fuzzy Logic Parameter OptimizerStudy: Using Autoencoder  A Novel Used Vehicles Price Prediction Model Based on Denoising Autoencoder With Convolution Operation