September 23, 2022

Fundamentals andApplications of WeakLearners
Muhammad Awais ShafiqueCentre d'Innovació del Transport (CENIT)
Universitat Politècnica de Catalunya (UPC)
Barcelona, Spain
1

Introduction• When predicting, the least one can do is Random Guessing• Weak Learner• “A weak learner produces a classifier which is only slightly more accurate thanrandom classification.”
Pattern Classification Using Ensemble Methods, pg 21, 20102

Introduction• A popular example is Decision Tree.
• Weakness can be controlled by the depth oftree.
• Weakest tree: only one node and binary
decision made on only one variable.

3

Introduction• Example: Your decision to participate in the summer school.
InterestNo

Yes
NewLearning

TeamNoNo

Yes

Yes

No
Yes

Enjoy
NoNo

Yes
Yes
Yes
4

Introduction• Example: Your decision to participate in the summer school.
NoNo

Interest

Yes
Yes

5

Introduction• Strong Learner• A strong learner produces a classifier that achieves arbitrarily good accuracy,
better than random guessing.
• For modeling tasks, we aim to develop a strong classifier that makes
predictions with good accuracy with high confidence.
• For instance, applying Support Vector Machines directly to the dataset.

6

Introduction• In short• Weak learners: Slightly better than random.
• Strong learners: Having good or even near-optimal accuracy.
• Are they equivalent?

YES
7

Boosting• A strong learner can be constructed from many weak learners.
• This became the basis for boosting methods

AdaBoost

XGBoost,LightGBM
8

Boosting• Develop a large number of weak learners for a predictive learning problem.
• Combine them in a way to achieve a strong learner.
• Weak learners: Easy to prepare but not desirable.
• Strong learners: Hard to prepare and highly desirable.

9Bagging vs. Boosting
Boosting• Start with one decision tree stump (weak learner) and “focus” on the samples it
got wrong.
• Train another decision tree stump that attempts to get these samples right.
• Repeat until a strong classifier is developed.
Less
underfittingMoreunderfitting
Boosting
10

Bagging vs. Boosting
Bagging• Train a number (ensemble) of decision trees from bootstrap samples of yourtraining set.
• After the decision trees are trained, we can use them to classify new data via
majority rule.
MoreoverfittingLessoverfitting

Bagging11

"Improving ridership by predicting
train occupancy levels"

12

Consider a Scenario

13

14

15

40%

56%

89%

72%

58%

Introduction

16

Survey• If future crowdedness levels are known 80% participants revealed that they willchange their departure time and/or route to ensure less crowded transport.
17

Studied Train Route (NSW, Australia)

Illawarra Line
From Central Station
to Cronulla Station18

Train Occupancy (Nov 2018 – Feb 2019)
Medium, 13%High, 3%

Low, 84%19

Addressing Imbalanced Data

Amount of Data (%)

1008060
40200
NormalDown Sampling
HighOver Sampling

MediumBothLow
20

Overall Accuracy

Sampling

Classifier
XGBRFSVM

Normal

0.941

0.956

0.951

Down Sampling

0.930

0.940

0.935

Over Sampling

0.958

0.959

0.936

Both0.955

0.955

0.939
21

Macro-averaged F1 Score

Sampling

Classifier
XGB

RFSVM

Normal

0.824

0.876

0.865

Down Sampling

0.831

0.864

0.844

Over Sampling

0.891

0.883

0.826

Both0.891

0.889

0.838
22

Challenges
Imbalanced Data: Biased learning leading to skewed results.
Attracted Occupancy: Predicted Crowdedness values would be affected bychanged travel behavior.

23

Predicted Crowdedness Level

Actual Crowdedness Level

Attracted Occupancy
24

Centre d’Innovació del Transport (CENIT)
C/ Jordi Girona, 1-3, C3, S120, 08034, Barcelona

www.cenit.es

A research group of:Annex• Precision =• Recall =!"#$ %&'()(*$!"#$ %&'()(*$+,-.'$ %&'()(*$!"#$ %&'()(*$!"#$ %&'()(*$+,-.'$ /$0-)(*$• Accuracy =!"#$ %&'()(*$+!"#$ /$0-)(*$!"#$ %&'()(*$+!"#$ /$0-)(*$+,-.'$ %&'()(*$+,-.'$ /$0-)(*$• F1 Score per class, F11 = 2×%"$1('(&2×4$1-..
%"$1('(&2+4$1-..

• Macro − Averaged F1 Score =,5!"#$ +,5%&'"() +,5*+,626