Package: skelo
skeloAn implementation of theEloandGlicko2rating systems with ascikit-learn-compatible interface.Theskelopackage is a simple implementation suitable for small-scale rating systems that fit into memory on a single machine.
It's intended to provide a convenient API for creating Elo/Glicko ratings in a data science & analytics workflow for small games on the scale thousands of players and millions of matches, primarily as a means of feature transformation in othersklearnpipelines or benchmarking classifier accuracy.MotivationWhat problem does this package solve?Despite there being many rating system implementations available (e.g.sublee/eloddm7018/Elo,rshk/elo,EloPy,PythonSkills,pyglicko2,glicko2,glicko) it's hard to find one that satisfies several criteria:A simple and clean API that's convenient for a data-driven model development loop, for which use case the scikit-learn estimatorinterfaceis thede factostandardExplicit management of intervals of validity for ratings, such that as matches occur a timeseries of ratings is evolved for each player (i.e. type-2 data management as opposed to type-1 fire-and-forget ratings)This package addresses this gap by providing rating system implementations with:a simple interface for in-memory data management (i.e. storing the ratings as they evolve)time-aware ratings retrieval (i.e.resolvinga player to their respective rating at an arbitrary point in time)scikit-learn classifier methods to interact with the predictions in a (more) typical data science workflowInstallationInstall via the PyPI packageskelousing pip:pip3installskeloAlternatively, a bleeding edge release is available from GitHub directly:pip3installgit+https://github.com/mbhynes/skelo.gitFurther reference information on the API is hosted onskelo.readthedocs.io.QuickstartAs a quickstart, we can load and fit anEloEstimator(classifier) on some sample tennis data:importnumpyasnpimportpandasaspdfromskelo.model.eloimportEloEstimatordf=pd.read_csv("https://raw.githubusercontent.com/JeffSackmann/tennis_atp/master/atp_matches_1979.csv")labels=len(df)*[1]# the data are ordered as winner/losermodel=EloEstimator(key1_field="winner_name",key2_field="loser_name",timestamp_field="tourney_date",initial_time=19781231,).fit(df,labels)The ratings data are available as apandas DataFrameif we wish to do any further analysis on it:>>>model.rating_model.to_frame()keyratingvalid_fromvalid_to0TimVann1500.0000001978123119790115.01TimVann1490.35094119790115NaN2AlejandroGattiker1500.0000001978123119790917.03AlejandroGattiker1492.5294781979091719791119.04AlejandroGattiker1485.41522819791119NaN...............8462TomGullikson1483.9145451979102919791105.08463TomGullikson1478.9347551979110519791105.08464TomGullikson1489.4005211979110519791105.08465TomGullikson1498.7570801979110519791113.08466TomGullikson1490.60000919791113NaNOnce fit, we can transform aDataFrameorndarrayof player/player match data into the respective ratings for each player immediatelypriorto the match>>> model.transform(df, output_type='rating')

               r1           r2
0     1598.787906  1530.008777
1     1548.633423  1585.653196
2     1548.633423  1598.787906
3     1445.555739  1489.089241
4     1439.595891  1502.254666
...           ...          ...
3954  1872.284295  1714.108269
3955  1872.284295  1698.007094
3956  1837.623245  1714.108269
3957  1837.623245  1698.007094
3958  1698.007094  1714.108269

[3959 rows x 2 columns]Alternatively, we could also transform a datafrom into the forecast probabilities of victory for the player"winner_name":>>> model.transform(df, output_type='prob')

0       0.597708
1       0.446925
2       0.428319
3       0.437676
4       0.410792
          ...
3954    0.713110
3955    0.731691
3956    0.670624
3957    0.690764
3958    0.476845
Length: 3959, dtype: float64These probabilities are also available using thepredict_probaorpredictclassifier methods, as shown below. What distinguishestransformfrompredict_probais thatpredict_probaandpredictreturn predictions that only use past data (i.e. you cannot cheat by leaking future data into the forecast), whiletransform(X, strict_past_data=False)may be used to compute ratings that "peek" into the future and could return ratings updated using match outcomes pushed (slightly) back in time to the match start timestamp. This is a specific convenience utility for non-forecasting use cases in which the match start time is a more convenient timestamp with which to index and manipulate data.>>> model.predict_proba(df)

           pr1       pr2
0     0.597708  0.402292
1     0.446925  0.553075
2     0.428319  0.571681
3     0.437676  0.562324
4     0.410792  0.589208
...        ...       ...
3954  0.713110  0.286890
3955  0.731691  0.268309
3956  0.670624  0.329376
3957  0.690764  0.309236
3958  0.476845  0.523155

[3959 rows x 2 columns]

>>> model.predict(df)

0       1.0
1       0.0
2       0.0
3       0.0
4       0.0
       ...
3954    1.0
3955    1.0
3956    1.0
3957    1.0
3958    0.0
Name: pr1, Length: 3959, dtype: float64Available Rating System EstimatorsThe models in this package are the following:EloEstimatorThis class is a pure python implementation of a standardElo rating system, without draws or homefield advantageGlicko2EstimatorThis class implements theGlicko2rating system, implemented using a light wrapper aroundglicko2Extended Usage ExamplesSynthetic DataWe can use a convenience utility for generating match data from players with normally distributed ratings.
The below example shows a simple usage of this utility:importpandasaspdimportmatplotlib.pyplotaspltimportskelo.utils.elo_dataasdata_utilsfromskelo.model.eloimportEloEstimator# Generate some sample constant ratings & match dataratings=data_utils.generate_constant_ratings(num_players=10,sigma=500)matches=pd.DataFrame(data_utils.generate_constant_game_outcomes(ratings,num_timesteps=100))# Fit the model using numpy arraysX=matches.values[:,:3]# player1, player2, match timestampy=matches.values[:,-1]# match outcomemodel=EloEstimator().fit(X,y)# Get a dataframe of the estimated ratings over time from the fitted modelratings_est=model.rating_model.to_frame()# Compare the ratings estimate over timets_est=ratings_est.pivot_table(index='valid_from',columns='key',values='rating')ts_est.plot()The estimated ratings will exhibit convergence profiles (players with extremal low or high ratings take longer to converge).
Please note that while the actual original ratings are unlikely to be determined by the fitting procedure, therelativedifference between the ratings should be preserved, within the noise band of the chosen value ofk(by default: 20)Example Tennis Rankingimportnumpyasnpimportpandasaspdfromskelo.model.eloimportEloEstimatorfromsklearn.metricsimportprecision_score# Download a dataframe of example tennis data from JeffSackmann's ATP match repository (thanks Jeff!)defload_data():df=pd.concat([pd.read_csv("https://raw.githubusercontent.com/JeffSackmann/tennis_atp/master/atp_matches_1979.csv"),pd.read_csv("https://raw.githubusercontent.com/JeffSackmann/tennis_atp/master/atp_matches_1980.csv"),],axis=0)# Do some simple munging to get a date and a match outcome labeldf["tourney_date"]=pd.to_datetime(df['tourney_date'],format='%Y%m%d')# For measuring classifier calibration linearity, it's better to have both true and false# labels in the dataset, so we relabel the order of (winner, loser) to just be (player1, player2)order_mask=(df["winner_name"]<df["loser_name"])df["p1"]=""df["p2"]=""df["label"]=1df.loc[order_mask,"p1"]=df.loc[order_mask,"winner_name"]df.loc[~order_mask,"p1"]=df.loc[~order_mask,"loser_name"]df.loc[order_mask,"p2"]=df.loc[order_mask,"loser_name"]df.loc[~order_mask,"p2"]=df.loc[~order_mask,"winner_name"]df.loc[~order_mask,"label"]=0returndfdf=load_data()player_counts=pd.concat([df["p1"],df["p2"]],axis=0).value_counts()players=player_counts[player_counts>5].indexmask=(df["p1"].isin(players)&df["p2"].isin(players))X=df.loc[mask]# Create a model to fit on a dataframe.# Since our match data has np.datetime64 timestamps, we specify an initial time explicitlymodel=EloEstimator(key1_field="p1",key2_field="p2",timestamp_field="tourney_date",initial_time=np.datetime64('1979','Y'),).fit(X,X["label"])#  Retrieve the fitted Elo ratings from the modelratings_est=model.rating_model.to_frame()ts_est=ratings_est.pivot_table(index='valid_from',columns='key',values='rating').ffill()idx=ts_est.iloc[-1].sort_values().index[-5:]ts_est.loc[:,idx].plot()This should result in a figure like the one below, showing the 5 highest ranked (within the Elo system) players based on this subset of ATP matches:Example Tennis Ranking - Elo ratings using thesklearnAPIWhile the ratings are interesting to visualize, the accuracy of the rating system's predictions are more important.
For determining the performance of a classifier, thesklearnAPI and model utilities provide simple tools for us.
Below we calculate the classification metrics of the Elo system using only the 1980 data, where each prediction for a match uses only the outcomes of previous matches:>>>fromsklearn.metricsimportclassification_report>>>mask=(X["tourney_date"]>"1980-01-01")>>>print(classification_report(X.loc[mask,"label"],model.predict(X.loc[mask])))precisionrecallf1-scoresupport00.6920.7040.698178210.7000.6890.6941792accuracy0.6963574macroavg0.6960.6960.6963574weightedavg0.6960.6960.6963574Finally, we should inspect the calibration curve of the classifier to verify its linearity (or lack thereof!):import matplotlib.pyplot as plt
from sklearn.calibration import calibration_curve

prob_true, prob_pred = calibration_curve(
  X.loc[mask, "label"],
  model.predict_proba(X.loc[mask]).values[:, 0],
  n_bins=10
)
plt.plot(prob_pred, prob_true, label=f"Elo Classifier, k={model.rating_model.default_k}", marker='s', color='b')
plt.plot([0, 1], [0, 1], label="Perfect Calibration", ls=":", color='k')
plt.xlabel("Predicted Probability")
plt.ylabel("Empirical Probability")
plt.legend()It's interesting to note that the calibration curve is not strictly linear, but rather has a slight but noticeable sigmoidal shape. If you plan on doing anything with the Elo predictions in aggregate, you may want to considercalibrating the classifier output.Example Paramter TuningWe can now also use thesklearnutilities for parameter tuning, similar to any other forecasting model.
The below example trains several instances of the Elo ratings model with different values ofkto find value that maximizes prediction accuracy during the ATP matches for 1980.
Please note that theEloModel.predict()method only uses past information available at each match, such that there is no leakage of information from the future in the model's forecasts.fromsklearn.model_selectionimportGridSearchCVclf=GridSearchCV(model,param_grid={'default_k':[10,15,20,25,30,35,40,45,50,]},cv=[(X.index,X.index[len(X)//2:])],).fit(X,X['label'])results=pd.DataFrame(clf.cv_results_)This should produce a result like the following:>>>results.sort_values('rank_test_score').head(5).T24168mean_fit_time0.1931560.1727730.4907190.1627930.045234std_fit_time0.00.00.00.00.0mean_score_time0.2690050.3048810.3364280.2886070.06117std_score_time0.00.00.00.00.0param_default_k2030154050params{'default_k':20}{'default_k':30}{'default_k':15}{'default_k':40}{'default_k':50}split0_test_score0.6787790.6771150.676560.6754510.675451mean_test_score0.6787790.6771150.676560.6754510.675451std_test_score0.00.00.00.00.0rank_test_score12344Example Tennis Ranking - Glicko2 ratings using thesklearnAPIThe Glicko2 implementation that we wrap has no hyperparameters to tune other than the 3-tuple to provide as theinitial_valuefor a player; namely the initial values of the rating, the rating deviation, and the volatility.
(However the persnickety reader should note there are quite a lot of magic numbers in both theGlickoandGlicko2papers that should probably constitute hyperparameters---these aren't available to tune simply because the available Glicko{1,2} implementations do not expose an interface that allows for tuning them. Beggars can't be choosers, as they say...)importnumpyasnpfromskelo.model.glicko2importGlicko2Estimatorfromsklearn.model_selectionimportGridSearchCVmodel=Glicko2Estimator(key1_field="p1",key2_field="p2",timestamp_field="tourney_date",initial_time=np.datetime64('1979','Y'),)clf=GridSearchCV(model,param_grid={'initial_value':[(1500.,200.,0.06),(1500.,350.,0.06),(1500.,500.,0.06),(1500.,750.,0.06),]},cv=[(X.index,X.index[len(X)//2:])],).fit(X,X['label'])results=pd.DataFrame(clf.cv_results_)We can now compare the best Glicko2 models with the Elo above and note that our test period forecasting accuracy has improved from 67.8% to 69.3%:>>>results.sort_values('rank_test_score').head(2).T32mean_fit_time0.2124660.217531std_fit_time0.00.0mean_score_time0.0688160.083714std_score_time0.00.0param_initial_value(1500.0,750.0,0.06)(1500.0,500.0,0.06)params{'initial_value':(1500.0,750.0,0.06)}{'initial_value':(1500.0,500.0,0.06)}split0_test_score0.6932040.692649mean_test_score0.6932040.692649std_test_score0.00.0rank_test_score12Development SetupIf you would like to contribute to this repository, please open anissuefirst to document the extension or modification you're interested in contributing.devscriptsThedevscript (and other scripts inbin) contain convenience wrappers for some common development tasks:Installing Dependencies. To set up a local development environment with dependencies, use:./devupThis will create a.venvdirectory in the project root and install the required python packages.Running Tests. To run the project tests with the virtual environment'spytest, use:./devtestArguments topytestcan be supplied to this script as well, to e.g. limit the tests to a particular subset.Packaging for PyPI. The package-building and upload process is wrapped with the following commands:./devpackage./devupload--test./devuploadCreating anRatingEstimatorThe available models extend theskelo.model.RatingEstimatorclasswhich implements thesklearnwrapper interface around askelo.model.RatingModelinstance.To create a new classifier, it's necessary to:Extend theRatingModelto implement the rating update formulas through the methods:evolve_rating(r1, r2, label), which computes a new rating given the players' previous ratings,r1andr2, prior to a match with outcomelabelcompute_prob(r1, r2), which computes the probability of victory of a player with ratingr1over a player with ratingr2classEloModel(RatingModel):def__init__(self,default_k=20,k_fn=None,initial_value=1500,initial_time=0,**kwargs):super().__init__(initial_value=float(initial_value),initial_time=initial_time)# Set all hyperparameters as explicit attributes, such that sklearn's CV utilities workself.default_k=default_kself.k=k_fnor(lambda_:default_k)defevolve_rating(self,r1,r2,label):exp=self.compute_prob(r1,r2)returnr1+self.k(r1)*(label-exp)@staticmethoddefcompute_prob(r1,r2):diff=(r2-r1)/400.0prob=1.0/(1+10**diff)returnprobExtend theRatingEstimatorto wrap the newRatingModelsubclass and specify the list ofRatingModelattributes that should be considered hyperparamters when dynamically building aRatingsModelin the estimator'sfitmethodclassEloEstimator(RatingEstimator):# Set this to the appropriate RatingModel:RATING_MODEL_CLS=EloModel# Include all tunable hyperparameter attributes of the RatingModel here:RATING_MODEL_ATTRIBUTES=['default_k','k_fn','initial_value','initial_time',]def__init__(self,key1_field=None,key2_field=None,timestamp_field=None,default_k=20,k_fn=None,initial_value=1500,initial_time=0,**kwargs):super().__init__(key1_field=key1_field,key2_field=key2_field,timestamp_field=timestamp_field,initial_value=initial_value,initial_time=initial_time,**kwargs)# These must be set as estimator attributesself.default_k=default_kself.k_fn=k_fnPlease note that aratingcan be anything, so long as it's convenient and can support the above call signatures to create a new rating object like-for-like from existing rating objects.
For example, ourEloModelimplementation uses a plainfloat, and theGlicko2Modeluses a 3-tuple for the 3 generative parameters for a player's rating.