fabiochiu/medium-articles huggingface dataset:
Forecasting Hard Coral Coverage with ARIMA
By: Michael Berk
Tags: Time Series Analysis, Statistics, Coral Reefs, Data Science, Time Series Forecasting

What’s the future of the world’s coral reefs?

In February of 2020, scientists at University of Hawaii Manoa released a study addressing this very question. The models they developed forecasted a 70–90% worldwide loss of coral by 2040. Even more alarming, they projected that “few to zero suitable coral habitats will remain” by the year 2100.

So, the future of coral doesn’t look great.

Interested in seeing these numbers firsthand, today we will develop our own forecasts of hard corals in the Caribbean. After restructuring the data, we’ll fit an extremely famous time series model: ARIMA. ARIMA has been popularized due to its simplicity and specific ability to fit time series data. Once we have a working model, we’ll develop a forecast.

Let’s jump right in.

Why do you need to aggregate?

We’ll start by taking a look at our raw data. In this case, we are going to be building a univariate model, so we’re only concerned with hard coral percent cover (i.e. the estimated percentage of hard coral on the sea floor).

Figure 1: Average daily % cover for hard coral on Caribbean reefs. Data source is Reef Check.

In the figure to the left, we have plotted the daily average of hard coral over time (blue). The y-axis shows the percent cover and the x-axis shows the date, ranging from 1997–2019. We also plotted a weighted linear regression line (red) to depict the overall trend.

Ok, seems straight-forward.

But if we try to interpret these data, we see a blue mess with a negative trend line; there appears to be little systematic movement. Moreover, according to the regression line, hard coral only decreased by around 4% over the past 22 years. That’s pretty hard to believe. So, as creative and skilled data scientists, let’s try some manipulations and see if we can develop a clearer picture.

First, it’s important to know how the data are organized. Unlike most time series datasets, these data were sampled at different locations around the Caribbean with few subsequent draws at the same site. Moreover, they are not equally sampled over time. So, to account for the above points, let’s aggregate the data by time.
How to run an Airbnb business in Sydney
By: Xu Hao
Tags: Time Series Analysis, Data Science, Syndey, Airbnb, Spatial Analysis

Part 3: What’s the pricing strategy?

I compared the available rate and average room price of the top 10 room amount areas with the rest. The high ten neighborhoods with the most significant room amount are Sydney, Waverley, Randwick, Manly, Warringah, Woollahra, North Sydney, Marrickville, Pittwater, and Leichhardt. They are the most famous waterside areas and account for 76.5% of Airbnb rooms in Sydney.

Figure 3: Time series analysis of average price and available rate

I find another support for choosing the waterside areas here. In the top 10 regions, despite the plentiful supply of rooms, the average price of the available room is about $320, the highest average price can reach $550, they are both nearly twice the rest 28 areas. The available rate is almost less than 30% for the top 10 and is lower than in other areas’ 50%. Waterside area means higher demand, higher price, lower vacancy rate. These benefits can offset the lack of experience for new beginners.

As for pricing strategy, the above figure has lots of information, and I had four takeaways:

Set the weekend price 3% to 5% higher than weekdays. Set the price around Christman and new year’s day about 1.5 to 1.8 times than usual. There are three times to increase your base price. They are the first week of March, June, and September. From later January to early March, it’s better to set your price the lowest of the year.

Because of high demand, around Christman and new year’s day, there is a price spike, the available room’s price is about 1.5 to 1.8 times as usual. This rate is even higher in the top 10 areas. The available rate is also the lowest of the year. But after the holiday, the price and the available rate return to normal quickly.

The available rate shows an apparent step-like distribution. From January to early March, Airbnb available is the highest of the year, and the price is the lowest. Then in the first week of Mar, Jun, and Sep, the available rate decreases dramatically, especially at the beginning of Mar and Jun, the average rate has a more than 10% decrease. The price follows the opposite trend.

After releasing the highest demand around new year’s day, Sydney comes into the hottest period of the year as a city in the southern hemisphere. I think that’s why Sydney has the highest available rate at that time.
Time Series Split with Scikit-learn
By: Keita Miyaki
Tags: Machine Learning, Time Series Analysis, Python, Split, Time Series Data

In time series machine learning analysis, our observations are not independent, and thus we cannot split the data randomly as we do in non-time-series analysis. Instead, we usually split observations along with the sequences.

Photo by Sonja Langford on Unsplash

We split data into training set and test set in everyday machine learning analyses, and oftentimes we use scikit-learn’s random splitting function.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y)

In time series analysis, however, we are not able to use this simple command, since observations in our time series datasets are not independent. The characteristics of time series data, such as autoregressive nature, trend, seasonality, or cyclicality, would not allow a random split to be valid. As a simple example, if your observations are autocorrelated, having an observation at time t in the training set and another observation at time t+1 in the test set would cause a trouble. A model which knows the former naturally predict the latter well. In that case, the test score will be too optimistic about the predictive power of the model.

We separate the test data from training data as a certain part of the end of the dataset. If you have observation over 10 years, for example, you may use first 7 years for training and the last 3 years for testing the model. The code is simple.

X_train = X[:int(X.shape[0]*0.7)]

X_test = X[int(X.shape[0]*0.7):]

y_train = y[:int(X.shape[0]*0.7)]

y_test = y[int(X.shape[0]*0.7):]

In this way we can train our models with less concerns about the validation. The same can be said about hyper-parameter tuning through cross-validation. The basic approach for that in non-time-series data is called K-fold cross-validation, and we split the training set into k segments; we use k-1 sets for training for a model with a certain set of hyper-parameters and measure the performance over the remaining set. We try it for k times over different combination of segments. For this scikit-learn’s GridSearchCV is handy. Yet, for the same reason stated above, we cannot use this convenient function for time series data.

Scikit-learn offers a function for time-series validation, TimeSeriesSplit. The function splits training data into multiple segments. We use the first segment to train the model with a set of hyper-parameter, to test it with the second. Then we train the model with first two chunks and measure it with the third part of the data. In this way we do k-1 times of cross-validation. Grid-search of hyper-parameter with TimeSeriesSplit requires some manual coding.
Bokeh Battles — Part 1: Multi Line Plots
By: Andrew Maguire
Tags: Bokeh, Time Series Analysis, Visualization, Python, Data Science

I’ve been trying to build a lot of standardized visualization functions in work recently and kinda still amazed at how horrible the whole experience has been.

I used to hate Tableau, then grew to love it, and so in my previous job i’d just point Tableau at the data and it would easily do 90% of what i’d need in a few clicks.

Now however, i’m doing much more visualizations in notebooks etc on pandas data so can’t really just point Tableau at it.

So as i build out these plotting functions for myself i’m going to try make them as generic as possible and stick them all in this repo and share any parts that might be interesting here.

Main things i had to Google my way out of and found tricky (hence the post):

Plotting multiple lines on same plot in bokeh and ensuring they all have their own hover tools and legend entries. Cycling through a color palette using a generator if you decide to throw lots of lines at it. Sharing axis on line plots when plotting them in a grid separate.

Multi Line Plots

First up is a multi-line plot of time series data.

interactive versions in nbviewer here

The make_data() function below makes some dummy data and puts it into a pandas dataframe.

Which we can just call like below:

df = make_data()

print(df.shape)

df.head()

Then we can define a function to plot all the numeric data in our df.

That’s it. I’ll probably add in more little examples like this as i build out my library of go-to plots.

p.s. Here is a Google Colab Notebook with everything that you should be able to just run out of the box.
Time Series Decomposition In Python
By: Billy Bonaros
Tags: Statsmodels, Time Series Analysis, Data Analysis, Python, Time Series Decompose

Time series decomposition is a technique that splits a time series into several components, each representing an underlying pattern category, trend, seasonality, and noise. In this tutorial, we will show you how to automatically decompose a time series with Python.

To begin with, let's talk a bit about the components of a time series:

Seasonality: describes the periodic signal in your time series.

Trend: describes whether the time series is decreasing, constant, or increasing over time.

Noise: describes what remains behind the separation of seasonality and trend from the time series. In other words, it’s the variability in the data that cannot be explained by the model.

For this example, we will use the Air Passengers Data from Kaggle.



import numpy as np

from statsmodels.tsa.seasonal import seasonal_decompose



#https://www.kaggle.com/rakannimer/air-passengers

df=pd.read_csv(‘AirPassengers.csv’)



df.head() import pandas as pdimport numpy as npfrom statsmodels.tsa.seasonal import seasonal_decomposedf=pd.read_csv(‘AirPassengers.csv’)df.head()

Image by Author

Firstly, we need to set as index the Month column and convert it into Datetime Object.

df.set_index('Month',inplace=True)

df.index=pd.to_datetime(df.index) #drop null values

df.dropna(inplace=True)

df.plot()

Image by Author

The Decomposition

We will use Pythons statsmodels function seasonal_decompose.

result=seasonal_decompose(df['#Passengers'], model='multiplicable', period=12)

In seasonal_decompose we have to set the model. We can either set the model to be Additive or Multiplicative. A rule of thumb for selecting the right model is to see in our plot if the trend and seasonal variation are relatively constant over time, in other words, linear. If yes, then we will select the Additive model. Otherwise, if the trend and seasonal variation increase or decrease over time then we use the Multiplicative model.

Our data here are aggregated by month. The period we want to analyze is by year so that's why we set the period to 12.

We can get each component as follows:

result.seasonal.plot()

Image by Author

result.trend.plot()

Image by Author

Also, we can plot every component at once

result.plot()

Image by Author

Wrapping up

Frequently, when looking at time series data it’s difficult to manually extract the trend or identify the seasonality. Fortunately, we can automatically decompose a time series and helps us have a clearer view of the components as It’s easier to analyze the trend if we remove the seasonality from our data and vise versa.
Understanding Partial Auto-Correlation
By: Sachin Date
Tags: Programming, Time Series Analysis, Statistics, Correlation, Data Science

Laying the foundation for PACF: The auto-regressive time series

There are many phenomena in which the past influences the present. The events of yesterday can be used to foretell what will happen today. When such phenomena are represented as a time series, they are said to have an auto-regressive property. In an auto regressive time series, the current value can be expressed as a function of the previous value, the value before that one and so forth. In other words, the current value is correlated with previous values from the same time series.

If a time series is auto-regressive it is often the case that the current value’s forecast can be computed as a linear function of only the previous value and a constant, as follows:

(Image by Author)

Here T_i is the value that is forecast by the equation at the ith time step. Beta0 is the Y-intercept of the model and it applies a constant amount of bias to the forecast. It also specifies what will be the forecast for T_i if the value at the previous time step T_(i-1) happens to be zero. Beta1 tells us the rate at which T_i changes w.r.t. T_(i-1).

The key assumption behind this simple equation is that the variance in T_(i-1) is able to explain all the variance expressed by all values that are older than T_(i-1) in the time series. It is as if T_(i-1) captures all the information associated with values older than itself.

But what if this assumption were not true? What if the variance in T_(i-1) is not able to explain all of the variance contained within T_(i-2)? In that case, the above equation will not be able to feed this unexplained portion of the variance from T_(i-2) into T_i, causing the forecast for T_i to go off the mark.

Fortunately it’s easy to fix this problem adding a term to the above equation as follows:

(Image by Author)

In this equation the extra term Beta2*T_(i-2) seeks to capture the variance contained in values that are older than T_(i-1) that could not be explained by the variance in T_(i-1). It feeds this balance amount of information directly into the forecast for today’s value T_i.

With the background established let’s build the definition and the formula for the partial auto-correlation function.
Forecasting Prices for Many Days Ahead
By: Leandro Rabelo
Tags: Machine Learning, Forecasting, Time Series Forecasting, Time Series Analysis

Within a company, there are several risk factors to be considered and managed. For instance, price changes where the manager has to decide the best moment to sell; product demand forecast to improve the inventory management. Another important factor to evaluate is forecasts about recurrent expenses since expressive values can impact the company’s working capital.

Usually, the forecasts on prices, demand, expenses, etc. are created taking into account the manager’s domain knowledge as well as essential information on the specific subject that influences decision making.

When predictive models are used, the data is separated between training and test set, the coefficients are generated in the training data and they are usually tested step by step in the test set. Thus, only one period is forecasted, then it updates with real values and so on until the end of the test set.

For this post an ARIMA model will be created, where the coefficients obtained in the training set will be tested in a slightly different way. 5 consecutive days will be forecasted and the series will be updated with real data (5 days in a row) and again, new forecasts will be created for 5 periods until the test data is over.

This kind of approach is interesting for those who want to create forecast models with several periods ahead, since it can easily be used to compare the forecasted extrapolations with real data, thus, verify what are the main characteristics of the model and what are the points that can be improved.

The main purpose of the model is to forecast the daily price for the next 5 days for 50kg bags of Crystal Sugar informed by Esalq, the data can be downloaded here (please, download the data from the Brazilian site, the English version doesn’t have the prices in Brazilian Reais (BRL), and the Jupyter Notebook with all codes can be found on my GitHub.

From now on, we will use the following steps extracted from my post: Basic Principles to Create a Time Series Forecast. (In case you are not yet familiar with the basis for creating these forecasts, I highly recommend you read it):

Split the series into training and test set,

Transform the series into stationary,

Search for correlated lags,

Create the model,

Plot the comparative graph on the training set,

Evaluate the model and the error in the test set,

Suggest possible improvements to the model.

Creating Training and Testing set

We have a price history of approximately 16 years and I will reserve the last 250 working days (approx 1 year) to test the model. There will be 50 comparatives of 5 days each, between the forecasts and current values. Note that all previous data will be used to train the model:
Reconstruct Google Trends Daily Data for Extended Period
By: Qingzong Tseng
Tags: Python, Google Trends, Time Series Analysis

We can have search trends data at daily resolution for any duration.

Scaling daily data by weekly trends could generated some artifacts.

Scaling using overlapped period is better as long as there are enough search activities during the overlaping period.

Code can be find here.

As google gained monopoly over the internet search, whatever we googled become another kind of measure of public interest over time. It has already been well explained by the Google News Lab and several articles have demonstrated data analytics based on the Google Trends, such as the cyclic pattern of ending a relationship with someone, predicting U.S. presidential race, etc.

The Limitations

To get most out of the search trends database, one can use the python module pytrends or R package gtrendsR. However, google currently limit the time resolution based on the query’s time frame. For example, query for the last 7 days will have hourly search trends (the so-called real time data), daily data is only provided for query period shorter than 9 months and up to 36 hours before your search (as explained by Google Trends FAQ), weekly data is provided for query between 9 month and 5 years, and any query longer than 5 years will only return monthly data.

My motivation into this subject was first inspired by the Rossmann competition in Kaggle where google search trends were used to predict sales number. I found it not so obvious to obtain the daily search trends and people used the weekly trends as surrogate. However, it is not ideal for any predictive model which necessitate precision at daily scale and real-time applications (as weekly data will only be available until the current week ends).

Comparing Methods for Reconstructing Daily Trends

Although people have already purposed methods to circumvent it (for example: here, and here). I was curious about how those methods compared to each other and matched best with the original daily trends data. Therefore, I used ‘iPhone’ as the keyword and reconstruct daily trends data over 35 months by the following three methods:
Notable Woman Owned Small Business Metrics in the GovCon Space
By: Paige Elmore
Tags: Woman Owned Business, Government Contracting, Metrics, Wosb, Women In Business

The Federal Government’s goal is to award at least 5% of all Federal Contracting Dollars to Woman Owned Small Businesses (WOSB and EDWOSB) each year.

In 2015, Congress made changes to the WOSB Federal Contracting Program that made it easier for qualified small businesses to participate in the WOSB program (2015 National Defense Authorization Act -NDAA) . You can see the impact of those programs in the following graph available on GovTribe:

2015: 297.9 Million

2016: 477.8 Million

2017: 708.6 Million

2018: 665.4 Million

2019: 770.4 Million

2020: 879.8 Million

Between 2015 and 2020, the Funding of Awards set aside for WOSB and EDWOSB grew by 295%.

What are the types of awards being set aside for WOSB and EDWOSB organizations?

In order to best answer this question, I created some reports to show me the NAICS Categories as well as the PSC categories associated with the Awards set aside for WOSB and EDWOSB over the last 5 years. For those that do not know, NAICS and PSC’s are the way the Government classifies spend. If you’re experienced in the private sector, these are similar to UNSPSC’s.

Top 3 NAICS:

541519: Other Computer Related Services 541611: Administrative Management & General Consulting Services 541512: Computer System Design Services

Top 3 PSCs:

All Other/ None Listed R499: Other Professional Services R408: Program Management/ Support Services

The majority of the funding for NAICS has gone to Technical Services, Consulting, and some General Contracting Services.

PSC categories had quite a few records that were missing a PSC value, but were shortly followed by Professional Service and Technology categories.

Who are the the WOSB and EDWOSB companies winning these awards?

When looking at the funding for awards over the last 5 years, there is a rather distribution across most organizations. The highest percentage of the work went to Chags Health Information Technology; which was only 2.8% of all funding. The top 5 Vendors combined only make up 10.6% of the entire spend set aside for WOSB and EDWOSB businesses over the last 5 years.

You can find a full list at the link above; however the Top 5 Vendors, their totals, and percentages are below:
How to value Bitcoin both in the long and short time (webinar)
By: Stefano Gianti
Tags: Webinar, Gold, Bitcoin, Metrics, Crypto

How to value Bitcoin both in the long and short time (webinar) Stefano Gianti Follow Mar 4 · 4 min read

Valuing Bitcoin over the Short and Long term is a webinar organized by Swissquote Bank with the 21Shares team. 21Shares is the world’s first digital asset, ETP issuer.

The combination of two investor-oriented entities has made this webinar a beneficial watch for investors looking into investing in the Bitcoin market.

This webinar assesses the possible returns based on an in-depth analysis of an investment strategy that focuses on Bitcoin and its performance.

The webinar is split into two sections. The initial part focuses on the short term evaluation of Bitcoin, and then it evolves gradually into the assessment of Bitcoin for investment over the long term.

www.swissquote.com/crypto

This webinar assesses the outlook of getting profitable returns based on an in-depth analysis of an investment strategy that focuses on investing in Bitcoin through the 21share Bitcoin trackers, the three indexes that 21Shares has, and a shorts fund that can help interested investors get a feel over the Bitcoin market.

This more than an hour-long webinar is comprehensively focused on helping the viewer understand all the multiple aspects of a portfolio strategy that targets investments in some version of Bitcoin investment.

An exciting thing disclosed in the webinar is that the Bitcoin options are no different from usual futures as they also have expiry dates. The webinar also clarifies the reasons behind the sharp fluctuations in Bitcoin at specific points, making investors use caution in investing. The overcorrections and over liquidations are discussed, and their causes are also explained.

As a short term strategy, it is useful to keep an eye on upcoming option expiry dates. If an expensive option is due to expire, it will create turbulence in Bitcoin prices worldwide, and price volatility will be seen.

The webinar focused on explaining how we can invest in Bitcoin by analyzing the holder information in the coinbase. The coinbase shows who is holding the bitcoin and who is buying, and who is selling. This information allows smaller investors to see which institutional buyers are getting into positions that can affect their pricing and will enable them to book gains over the short term.

While we have done our best to summarize this information-packed hour into this article, to get the most of it, watch the entire webinar here:
Are all CNNs created equal?
By: Robert Geirhos
Tags: Deep Learning, Human Perception, Human Behaviour, Metrics, Recurrent Neural Network

Motivation: erring is telling

So you want to train a neural network to distinguish puppies from people. Maybe you’d like to train a system that opens the door when your little puppy arrives but keeps strangers out, or you are the owner of an animal farm where you want to make sure that only people can get into the house.

In any case, you take your favourite CNNs (say, ResNet-152 for performance and AlexNet for good old times’ sake) and train them on a puppies-vs-people dataset scraped from the web. You are relieved to see that each of them reach about 96-98% accuracy. Lovely. But does similar accuracy imply similar strategy?

Well, not necessarily: even very different strategies can lead to very similar accuracies. However, those 2–4% that the networks got wrong carry a lot of information about their respective strategies. Suppose AlexNet and ResNet both made an error on the following image by predicting “person” instead of “puppy”:

Photo by Charles Deluvio on Unsplash

Now making this error would be just as sad as this dog looks like, especially since many other puppies were recognised perfectly well, like these here:

You probably know what puppies look like, but aren’t they cute?! Photo by Bharathi Kannan on Unsplash.

Looking at some more images that the networks failed to recognise, you start forming a suspicion: Could it be the case that both networks implemented the classification strategy “whatever wears clothes is a person”? You’ve always suspected AlexNet to be a bit of a cheat — but what about you, ResNet? Is it too much to ask for a bit more depth of character from someone with 152 layers?

Closer inspection confirms: conversely, the networks also misclassified a few people as “puppies” —we leave it to the reader’s imagination how these images may have looked like if the model’s decision strategy relies on the degree of clothing.

Erring is telling, and we can exploit this property: if two systems (e.g. two different CNNs) implement a similar strategy, they should make errors on the same individual input images — not just a similar number of errors (as measured by accuracy) but also errors on the same inputs: similar strategies will make similar errors. And this is exactly what we can measure using error consistency.

Introduced in our recent paper, trial-by-trial error consistency assesses whether two systems systematically make errors on the same inputs (or trials, as this would be called in psychological experiments). Call it an analysis based on trial and error if you like.

Leaving the hypothetical toy dataset (puppies vs. people) aside, how similar are the strategies of different CNNs trained on a big dataset (ImageNet)? And are they similar to human errors on the same data? We simply went to the animal, pardon, model farm and evaluated all ImageNet-trained PyTorch models to obtain their classification decisions (correct responses vs. errors) on a dataset where we also have human decisions for comparison. Here’s what we’ve discovered.
PyTorch Lightning: metrics
By: Justus Schock
Tags: Deep Learning, Pytorch, Metrics, Lightning, Tutorial

With PyTorch Lightning 0.8.1 we added a feature that has been requested many times by our community: Metrics. This feature is designed to be used with PyTorch Lightning as well as with any other PyTorch based code. In this blog post, we’ll explain what Metrics is and how you can get started

Disclaimer: This Post is no longer the recommended way to use metrics. We redesigned it completely and outsourced it as a separate package. See this GitHub or this Medium link for reference.

That means, that the implementation differs now and more closely follows the idea of a torch.nn.Module than before, but the following chapters as of why a Metric is important is still relevant.

What’s a metric and why do I need this?

The purpose of metrics in general is to allow you to monitor and quantify the training process. If you are using more advanced techniques like learning-rate scheduling or early stopping, you are probably using metrics for this. While you could also use losses for this, metrics are preferred since they represent the training goal better. This means, that while the loss (like cross-entropy) pushes the network’s parameters into the correct direction, metrics can show some additional insights on network behavior.

In opposite to losses, they also don’t have to be differentiable at all(in fact many of them aren’t), but some of them are. If the metric itself is differentiable and it is implemented using pure PyTorch it can also be used to backpropagate over it and use it for whatever fancy research you want to do.

In the mathematical sense, a metric is defined as a distance between each pair of elements in a set. While this definition also holds for the deep-learning understanding of a metric, some of the mathematical constraints of a metric must not be fulfilled here, since we don’t need them to be symmetric in all cases and they also don’t necessarily have to hold the triangle inequality.
Avoid R-squared to judge regression model performance
By: Kevin Dunn
Tags: Model Interpretability, Metrics, Regression, Least Squares

Avoid R-squared to judge regression model performance

Summary

R² can be calculated before even fitting a regression model, which doesn’t make sense then to use it for judging prediction ability. Also you get the same R² value if you flip the input and output around. Again, this is nonsensical for a prediction metric.

The intention of your regression model is the important factor for choosing an appropriate metric, and a suitable metric is probably not R².

This article explains which better alternatives exist: the standard error, confidence intervals and prediction intervals.

1. Why do we build linear regression models?

Historical data is collected and a relationship between the input 𝑥 and the output 𝑦 is calculated. This relationship is often a linear regression model, written as 𝑦=a+𝑏𝑥, where the intercept is a and the slope is b.

The purpose is most often to use that calculated relationship and make a prediction of some future output, called 𝑦̂, given a new input: 𝑦̂=a+𝑏𝑥.

Here are the two most common reasons for building a linear regression model:

to learn more about the relationship between input and output; by far the most common usage is to get predictions of the output, based on the input.

Let’s look at each of these in turn.

Learning more about the relationships

The coefficient, 𝑏 of the linear regression 𝑦=a+bx, shows what the average effect is on the output, 𝑦, for a one unit increase in the input 𝑥. This is called “learning about our system”.

The slope and intercept from a regression model provide learning, or insight, into the system. The figure also shows the terminology used here. [Source: own work]

For example, if you built a regression model between 𝑥=temperature measured in Celsius of your system (input) and the 𝑦=pH (the output) you might get a regression model of: 𝑦=4.5+0.12𝑥, from which you learn two things:

that every 1 degree increase in temperature, leads, on average, to an increase of pH by 0.12

that the expected or predicted pH when using a temperature of 𝑥=0 degrees Celsius, is an output pH of 4.5 units.

But consider two cases: what if I told you the R² of this model was 0.2, or it was 0.94. How does this change your learnings? We will come to this in the next section, where we understand a bit more what R² is measuring.

Using the model’s predictions

This scenario is the one most people are familiar with. Continuing the above, it is asking what the predicted pH would be for a given new input value of temperature, 𝑥. For example, at a new temperature that we have never used before of 13°C, we expect an output pH of 4.5+0.12×13=6.06 pH units.

And again, what value do we attach to such a prediction when the model has R² around 0.2, or if the model has R² of 0.94?
6 Common Metrics For Your Next Regression Project
By: Hien Vo
Tags: Machine Learning, Statistics, Metrics, Regression, Data Science

For the following sections, y is the true value, y-hat is the predicted value, n is the number of test instances, and i goes from 1 to n. Also, all metrics are evaluated on an unseen test set.

Mean Absolute Error

Image by Author. Generated by latex2png

Mean absolute error is a very intuitive and, therefore, popular metric. It is simply the average distance between predicted and true values. To avoid errors canceling one another out, we take the absolute of every error that we compute. The best model is usually the one with the lowest MAE. However, there are a few features to consider when picking MAE as your metric.

Although it is straightforward to interpret, MAE has a few disadvantages. For example, it doesn’t tell you whether your model tends to over-estimate or under-estimate since any direction information is destroyed by taking the absolute value. Also, the metric can be insensitive to large outliers. Take a look at the example below.

Image by Author. Generated by latex2png

On the left, the model is off by a bit here and there. However, on the right, the model misses the marks at the tail end by a much wider margin while being perfect at the beginning and in the middle. Yet, the MAEs for both are the same. If you decide to use this metric, it’s a good idea to plot the errors to see any outliers like case 2. All in all, if you want a metric that penalizes large errors, you’ll have better luck somewhere else.
Games and Goals
By: Kyle Johnson
Tags: Metrics, Jiu Jitsu, Data Science, Analytics

Games and Goals

Photo by Jonathan Borba on Unsplash

Brazilian jiu jitsu is undergoing a renaissance. With the advent of YouTube and social media, new techniques are instantly shared and iterated on. The entire community is taking part in a massive refining process. The discipline and the sport, though, are two separate things.

Perhaps the most succinct definition of the discipline is elite coach, John Danaher’s:

Brazilian Jiu-Jitsu is a grappling-based martial art whose central theme is the skill of controlling a resisting opponent in ways that force him to submit.

Danaher has played a huge hand in the recent evolution—he knows what a proficient grappler looks like. What’s the essence of grappling proficiency though? Is there an objective measure of how good any given grappler is in achieving these ends?

The short answer is no. There are practical limits to just how “real” any combat sport can be. The most effective way to control a person might actually be to gouge an eye out or to try to rip their ear off. We choose not to emphasize these techniques, though, for practical purposes (to put it lightly).

So because we can’t directlytest and measure the actual thing, we test and measure something close enough to it. A proxy for Danaher’s stated definition. We measure our progress by gamifying jiu jitsu and making it sport.

Sport jiu jitsu is our best “metric” for an athlete’s proficiency. Excessively violent or debilitating moves that would render an athlete permanently disabled have been removed to allow practitioners to train regularly. The ruleset encourages them to train in a sustainable way. It’s slightly off-center from measuring the true thing, but it’s close enough. The goal is to set up the rules in a way that ensures that an athlete who becomes a world champion is certainly among the best at controlling an opponent in a way that leads to submission.
Contribution Metrics Working Group
Tags: Collaboration, Art, Metrics, Community, Governance

Session 2

Not that we say so ourselves, but these conversations are getting deeper, more satisfying, and with more concrete insights.

In our second meeting, Bea presented a table of needs, motivations, and incentives.

We discussed the differences between them, how they relate to one another, and the patterns that emerge.

Take a look at the insights the group arrived at in the video:

Here are some links and notes that were shared in the chat.

Sparrow: The Surprising Truth About What Motivates Us from Daniel Pink, whom we quote extensively in our paper.

An apropos musical interlude from Sam:

And one from Ilan:

We were talking about the need for qualifications when the need to do something for a community implies safety issues, such as an engineer who wants to build bridges or buildings.

James: Does the bridge analogy break down as building bridges, at least in the UK requires some level of inspection, regulation, and controls which ensure that regardless of the motivation of the builder, it is still safe for everyone?

Speaking of incentives, they are basically a way to account for quantifiable value.

Naj: this is the way for the market to put a price on our needs.

There is a concept called Negotiation with self which helps us decide what we want to do next in life and what we value. Then, there is the concept of negotiation with society, where we decide what is important for us to see in society and struggle with society for that. Next comes Objective Conditioning which is how circumstances lead us to our current condition leading to unmet intrinsic needs where the market comes in to put a price on that need.

This book might help in understanding a few concepts related to this: https://www.amazon.com/Ego-Tunnel-Science-Mind-Myth/dp/0465020690

To refresh your mind, here is the Incentive Framework chapter on The Invisible Economy.

And here is the Octalysis gamification framework by Yu-kai Chow.

Lots of food for thought… See you in two weeks!
Russell’s Comm 445 Blog: Week 6
By: Russell Hoffman
Tags: Communication, Twitter, Wiu, Comm445, Metrics

This week, I was assigned to evaluate the reach of a social media account. I was allowed to choose one of my own or one from another person/company. I decided to go with the Twitter account for myself, as it would only allow a twitter account I had the email and password for. Plugging my information into the website resulted in a score sheet for my twitter account.

The metrics that were available in my free trial included a section for audience, publishing, and conversations within the last 30 days. Underneath the audience section was a box for followers (how many I have) and quality (how many responses I receive on posts). Then underneath the publishing section was a box for activity (how frequently I post), engagement (how much I interacted), and quality (which was how many likes and retweets I received). Lastly, the conversation section only had one box underneath labeled activity (which is how many conversations I had).

This tells us how much traction and interaction a Twitter account has. As is clearly evident on my Twitter account, I have a small number of followers with little interaction between them. That’s mostly because I use my account to follow creators that I like and see what their next project is. Overall, I believe that this is very useful for someone who works directly in the social media field and would find it helpful to see where they are doing well and where they falling behind and address the issues accordingly.
Part 1: How to Collect and Store Sustainability Data
By: Eliza Erskine
Tags: Small Business, Sustainability, Restaurant Business, Data Collection, Metrics

WHAT

Sustainability “data” includes how much waste, water or energy you are using, how much time you are allocating to sustainability efforts and how much money you are saving or spending on efforts. I recommend collecting two types of data: quantitative and qualitative. Collecting this data is a way to show progress, report to investors, and share with stakeholders. Quantitative data is “collected” for you in the form of bills and usage calculations. Qualitative data includes customer and employee feedback and takes conscious effort to organize.

HOW

The best way to collect and store data is a way that works for you.

Quantitative collection includes computable numbers or figures. Store data categorized by type: waste, water or energy. Put both money spent and the unit used into the spreadsheet. I find it easiest to store this in Excel sheets. It can be easy as creating columns of each item and then using rows across for the unit (months, usually).

Enter in the data each month for each indicator as you get your bill. It’s easiest to do this a little bit at a time instead of taking time to do it all at the end of the year. Do it as you pay your bills. Putting it in monthly also allows you to see changes and improvements month over month.

Collect qualitative data through surveys, customer comments, and writing down your own feelings or thoughts about sustainability. Employee complaints, struggles and interests are all valuable qualitative data. With each sustainability initiative, take time to collect employee feedback and ask them for input. Save that input into a central place. Qualitative data should also include rough estimates of time spent completing initiatives, changing items, or other related time.

REAL LIFE

Restaurateur:

· Tracks water, waste and energy into spreadsheets as part of monthly bill pay/financial reconciliation. One section for money spent, one section for units used

· 4x annual staff employee survey- store in word document

· Open drop box of employee feedback — retrieved monthly and stored in a word document

Retail Store:

· 10 minutes each week to jot down sustainability ideas and ask for employee feedback from 4 employees each month

· One sheet to show money spent on sustainability initiatives, including LEDs and wind protection for door

· One excel sheet to track bills and units used for electricity
3 metrics to make sure your app is delivering the best experience to its users.
By: Abdo Shalaby
Tags: Google Analytics, App Analytics, Metrics, Product Analytics, Product

3 metrics to make sure your app is delivering the best experience to its users.

Photo by Austin Distel on Unsplash

In my last post, I have introduced a simple 4-steps framework to organise your thinking of tracking your App. In this post, I want to share with you three metrics that I found helpful for product teams to monitor App health status.

Now, let’s discuss the metrics..

1- Usage: This metric is to answer how many users use the App flows/features/events. The absolute number is beneficial to know, but comparing it to benchmarks such as the previous periods or certain targets gives context to the number if it is good or bad. i.e. in Spotify App example, one event that you want to follow is upgrade_to_premium (How many users with free account upgrade to paid account). how this number is compared to the targets, and previous periods.

Another benefit for monitoring this metric is to see if certain features are used as expected. The usage trend may shows to you if your App users think of the features as you do, or if they give more weight to other features different that yours.

usage trend of upgrade_to_premium event.

2- Success rate: This metric is the percentage of users who achieved what they want from the App successfully, with no technical issues. i.e. what percentage of App users succeed to upgrade_to_premium with no technical issue. Our hope that this percentage is 100%, but you might be surprised that this usually not the case, and % of users get technical errors for some reason or another. You can track this metric by collecting success parameter with your tracked events.

success is tracked as an event parameter.

3- Completion rate: This metric is to measure how smooth are the App flows. It’s the ratio of users who complete the flow compared to the ones who start it. This should tell us where do users drop the most from the flow. One trick to be able to track this is to have consistent naming convention when naming events where the event position in the flow is mentioned in the event name. i.e. any event that has _start on it, is an event that start a flow, and any event that has _end or _complete on it means that it is the last event that should fire in the flow.

_start & _end suffix is added to the event.

Summary

In this post, I discussed 3 metrics that product teams can use to monitor App health status. The first is usage trends, and it aims to tell you which features are popular and which are not.

The second metric is success rate that aims to tell you if the user get any error while trying to achieve tasks in your app or digital product.

The last metric that we discussed is the completion rate and this metric tells you how smooth is your flows, and if users drop from it or not.

Thanks for reading!
Python Feature Importance detection
By: Nhu Hoang
Tags: Machine Learning, Feature Engineering, Metrics, Feature Importance, Data Science

What is included in Feature Importance? ✋

There are several ways to express Feature Importance. It can be score representing the features’ relevancy (using algorithm-based Feature Importance, such as tree-based Random Forest, XGBoost,..), dimension reduction from original features to synthesize all the information to a lower dimension (like Autoencoder or PCA), or domain knowledge selection (aka selection based on your own knowledge). With the variety of choices like this, which one should we choose? Even when we choose to use PCA, selecting the right number of remaining features seems to demand great exertion. Don’t you think we should go with the Feature Importance score or testing everything and see which performs the best?

Yes, it’s correct, testing everything is the accurate answer. From my experience, I usually set the baseline as the original model without any selection, then try PCA with different drop-out ratios and check the metric. Then drop original features (mainly because I want to limit the amount of collected data needed, as well as optimize the training time while maintain or improve the performance). So Dimension Reduction is my first choice and the Feature Importance is the next essential step. But using only the Feature Importance score is hardly a good choice. Let me show you why.

Why not only Feature Importance score alone 🌓

Feature Importance score here means any score generated from the trained model representing the weight or relevancy of the features to the prediction of the target feature.

Below are the feature importance scores of Random Forest calculated based on RandomForestRegressor model (detail of formula is here), selecting feature with SelectFromModel (detail), and permutation score (detail of formula is here). Data used for demonstration is California housing in sklearn.