The 20th Summer School of Behavioral Modelling
September 17, 2021

Behavioral Models Based on Weak Learners
using Multiple Sensors
Muhammad Awais Shafique, PhD
Severo Ochoa Postdoctoral Researcher
Center for Innovation in Transport (CENIT), Barcelona, Spain

Introduction• When predicting, the least one can do is Random Guessing• Weak Learner• “A weak learner produces a classifier which is only slightly moreaccurate than random classification.”
Pattern Classification Using Ensemble Methods, pg 21, 2010

• Weak Classifier• A classifier that achieves slightly better than 50% accuracy.
2

Introduction• “For binary classification, it is well known that the exact requirement
for weak learners is to be better than random guess.”• “Notice that requiring base learners to be better than random guess
is too weak for multi-class problems, yet requiring better than 50%
accuracy is too stringent.”
Ensemble Methods, pg 46, 20123

Introduction• A popular example is Decision Tree.
• Weakness can be controlled by the depth of tree.
• Weakest tree: only one node and binary decision made on only onevariable.
• “Because boosting requires a weak learner, almost any techniquewith tuning parameters can be made into a weak learner. Trees, as itturns out, make an excellent base learner for boosting.”
Applied Predictive Modeling, pg 205, 20134

Introduction• Strong Learner• A strong learner produces a classifier that achieves arbitrarily goodaccuracy, better than random guessing.
• For modeling tasks, we aim to develop a strong classifier that makes
predictions with good accuracy with high confidence.
• For instance, applying Support Vector Machines directly to thedataset.

5Introduction• In short• Weak learners: Slightly better than random.
• Strong learners: Having good or even near-optimal accuracy.

• Are they equivalent?

YES
6

Boosting• A strong learner can be constructed from many weak learners.
• This became the basis for boosting methods

AdaBoost

XGBoost7

Boosting• The goal of boosting ensembles• Develop a large number of weak learners for a predictive learning
problem.
• Combine them in a way to achieve a strong learner.
• Weak learners: Easy to prepare but not desirable.
• Strong learners: Hard to prepare and highly desirable.

8

Bagging vs. Boosting
Bagging• Train a number (ensemble) of decision trees from bootstrap samplesof your training set.
• After the decision trees are trained, we can use them to classify new
data via majority rule.
MoreoverfittingLessoverfitting

Bagging9

Bagging vs. Boosting
Boosting• Start with one decision tree stump (weak learner) and “focus” on thesamples it got wrong.
• Train another decision tree stump that attempts to get these samplesright.
• Repeat until a strong classifier is developed.
Less
underfitting

Moreunderfitting

Boosting
10

Paper for Discussion
Hak Lee, E., Kim, K., Kho, S.Y., Kim, D.K. and Cho, S.H., 2021. Estimating
Express Train Preference of Urban Railway Passengers Based on
Extreme Gradient Boosting (XGBoost) using Smart Card
Data. Transportation Research Record.
Key Points• XGBoost vs. MNL• SMOTE (Synthetic Minority Over-sampling Technique)• SHAP (SHapley Additive exPlanation)
11

Introduction• Express strategy on the urban
railway since 2009.
• Line 9-first private subway in Seoul
to introduce express trains.
• Local and express trains both on thesame railway.
• Local Train – 30 stops (100 min)
• Express Train – 13 stops (60 min)12

Collected Data

13

Collected Data

14

Pre-Processing• One day Smart Card Data and Train Log Data integrated.
• Passenger’s boarded train estimated.
• Other train generated as unselected alternative.

15

16

Pre-Processing

17Pre-Processing• Imbalanced data
• SMOTE (Synthetic Minority Over-sampling Technique) was used.
• SMOTE uses each data point of a minority class and generates newsamples along the line joining them to their k-nearest neighbors.
• With XGBoost we don’t need to worry about multicollinearity.
• XGBoost trained on 85% randomly selected data.
• Various hyperparameters were tuned.

18Performance Measure

19Performance Measure

20SHapley Additive exPlanation

21

SHapley Additive exPlanation

22

SHapley Additive exPlanation

23

SHapley Additive exPlanation

24

Concluding Remarks• XGBoost performs better than MNL.
• SMOTE can be conveniently used to address imbalanced data.
• SHAP can be used to understand the impact of each variable.

25

26