Triple-O for SHL Recognition Challenge: An Ensemble Framework forMulti-class Imbalance and Training-testing Distribution Inconsistency by OvO
Binarization with Confidence Weight of One-class Classification
12Jinhua Su ,Yuanyuan Zhang

1Renmin University of China, Beijing, China.

2Beijing Baixingkefu Network Technology Co., Ltd., Beijing, China.

Background
SHL Recognition Challenge provides phone sensor data forrecognizing eight modes of locomotion andtransportation(activities). Most motion sensor data isuser-dependent, so the model’s generalization abilityrequires various data at the user level. The SHL dataset iscollected only by three users, and category imbalance anddistribution gap between training data and testing data
can be significant obstacles for this challenge.
• There is no apparent advantage of the deep and heavymodel compared to some naive but robust machinelearning methods.

SolutionResults

This paper puts forward a new ensemble framework called
triple-O, using OvO binarization and one-class classification. The results show that the OvO binarization ensemble gets better results on the hard-to-distinguish class thanre-sampling and re-weighting.

• With re-sampling or other data imbalance methods,
models still give the worst prediction for the minor
category.
• Overfitting is common; most feeling-good models get alower ranking. The explicitly underfit model won firstplace.
• Traning only on the training dataset makes many
participants doubt life. Finetuning on the validation
dataset works.
SHL challenge also notices that problem and provides
user-independent sensor data in 2021[1].

Challenge
Category imbalance and distribution inconsistency are still
the obstacles for the SHL-2021 Challenge.

Splitting the multi-class classification task into several
OvO(one v.s. one) binary-class classification gives a solution to low intra-class separability. Indeed, [2] has done theexperiments on the relatively simple task for this proposal.
The results show OvO methods did outperform the OvR
methods, especially for the imbalanced situation.
To apply the most straightforward One-class Classification,
we calculate the centre of binary data collections. Furthermore, select 90% quantile of the distance between data andcentre as the radius r of the sphere. The above OvO voting
procedure is to calculate the sum of probability of each binary
meta-learner. We estimate 28 spheres(decision boundary forour one-class classifier) for the 28 meta-learner of the OvO
ensemble and use the following confidence equation(shownin 1) to get a weighted probability for final prediction.
(d−r)2− r2 ),d > r(1)
conf idence = exp
Triple-O applies the OvO method to tackle the multi-classimbalance problem caused by the softmax method. Experiments show its efficiency compared to other data imbalancesolutions, like re-sampling and re-weighting. What is more,another challenge of this dataset is the training and testing
distribution inconsistency. To tackle this problem, we use
a one-class classification method to detect the outlier fromthe training distribution. The predicting confidence can beused as the weight of meta-learner for soft voting.
The distribution inconsistency in user-level andcategory-level arise a domain adaptation problem forchallenge participants. The concept map is shown in theabove figure. Different user has different phone-using
manners at the same activity. Furthermore, different
locomotion and transportation activities depend on thereal-world transportation conditions and user commuting
habits, which indicates that the category ratio of one user
may vary during weekdays and weekends. Multi-modal
distribution in different levels makes recognition
challenging. Facing the same problem, face recognitionuse centre loss and marge loss to realize domain
adaptation. Lack of user category labels, it is hard to
apply centre loss for intra-class compactness directly.
However, inter-class separability can be integrated for theactivities recognition task. This paper is aimed to alleviate
the above problems drawing lessons from the research offace recognition

We split the validation dataset into 80%
validation-training part and 20% validation-testing partand directly train the model on the validation-training
and evaluated it on the validation-testing part. This
paper focuses on solving the data imbalance problem forthe multi-class classification problem. The results ofapplying different learning strategies on the meta-learnerare shown in the following table. Directly training on thevalidation-training part still performed relatively worse in
the prediction of Run and Bus. We are applying there-sampling and re-weighting methods whose results
show an improvement on the Run and Bus. The F1 forother activities also drops, which is reasonable that theprevious give a fairly good prediction for the majority.
The mean of all activities F1 score(equals to macro-F1)
do improve in a small amount. Using the binarization
ensemble will enhance the total parameters; the final
results appear to be overfitting. By looking up thetraining evaluation, 96.2 of macro-F1 indicates theoverfitting is not severe.
F1(%)
12345678
Direct93.5 91.8 67.4 89.4 87.3 71.2 92.7 91.7
Re-Sample 90.5 87.4 74.3 89.4 87.9 75.6 92.3 91.5
Re-Weight 89.2 88.4 73.9 90.1 86.8 77.2 91.8 92.3
OvO
97.6 92.8 84.7 99.9 94.1 85.2 99.3 98.3
Tirple-O
87.6 88.8 81.7 92.3 90.2 88.2 85.3 87.3
OvO+Train 96.4 91.5 80.3 99.9 92.7 81.1 98.3 97.4
Evaluation on different learning strategies on validation
dataset Notes: RF is short for random forest.
The results show that every type of activity becomesworse and worse. It is hard to claim the failure of
One-class classification for limited data and lack ofhyperparameter tuning. If the radius is too small,
one-class classification will limit the model generalization
ability. On the other hand, whether the sphere is asuitable boundary for data has not been explored.
123 45678
1 4114 88 0 60056 0
2 29 4572 0 021 028 0
3 0
100 204 00000
4 000 2082 00005 92 30 0 03516 266 0145
6 38 486 0 001784 40 07 000 0003487 08 070 0 00003999Finally, add the training-training dataset for the OvO
ensemble pretraining to make the model more robust.
The confusion matrix of this OvO ensemble evaluated on
the validation-testing is shown in the above table.

Conclusions
This ensemble uses the cascade framework, which is creativeand needs to be further researched. The OvO ensemble plus
One-class classification weighting pipeline is named Triple-O.
Triple-O can be plug-and-play and pave the way for exploring complex model structures. To our best knowledge, thismethod has not been further explored. The most relevant
work we can find is [3].

For SHL Recognition Challenge, the binarization strategy
works on the setting of distribution inconsistency andclass imbalance. The OvO ensemble method explicitly
outperforms in contrast to the re-sampling and re-weightmethods. To further tackle distribution inconsistency, wetry the One-class classification to re-weight thecontribution of each meta-learner to the final prediction.

References[1] Lin Wang, Hristijan Gjoreski, Mathias Ciliberto, Paula Lago, Kazuya Murao, Tsuyoshi Okita, and Daniel Roggen. Summary of the sussex-huawei locomotiontransportation recognition challenge 2021. In Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing andProceedings of the 2021 ACM International Symposium on Wearable Computers, 2021.
[2] Mikel Galar, Alberto Fernández, Edurne Barrenechea, Humberto Bustince, and Francisco Herrera. An overview of ensemble methods for binary classifiers in
multi-class problems: Experimental study on one-vs-one and one-vs-all schemes. Pattern Recognition, 44(8):1761–1776, 2011.
[3] Dong Chen, Xiao-Jun Wang, Changjun Zhou, and Bin Wang. The distance-based balancing ensemble method for data with a high imbalance ratio. IEEE Access,
7:68940–68956, 2019.

n:oitamfornit
caContSuauhni
Ju.cnde.
cur@chasesu nhuaSu
Ji:buhti
GLATEX TikZposter