markhneedham/blog huggingface dataset:
Strava: Calculating the similarity of two runs
Date: 2018-01-18 23:35:25
Learn how to compare runs from the Strava API using the Dynamic Time Warping algorithm.

I go running several times a week and wanted to compare my runs against each other to see how similar they are.

I record my runs with the https://www.strava.com/[Strava] app and it has an https://strava.github.io/api[API] that returns lat/long coordinates for each run in the https://strava.github.io/api/#polylines[Google encoded polyline algorithm format].

We can use the https://pypi.python.org/pypi/polyline/[polyline] library to decode these values into a list of lat/long tuples. For example:

[source,python]
----

import polyline
polyline.decode('u{~vFvyys@fS]')
[(40.63179, -8.65708), (40.62855, -8.65693)]
----

Once we've got the route defined as a set of coordinates we need to compare them. My Googling led me to an algorithm called https://en.wikipedia.org/wiki/Dynamic_time_warping[Dynamic Time Warping]

____
DTW is a method that calculates an optimal match between two given sequences (e.g. time series) with certain restrictions. The sequences are "warped" non-linearly in the time dimension to determine a measure of their similarity independent of certain non-linear variations in the time dimension.
____

The https://pypi.python.org/pypi/fastdtw[fastdtw] library implements an approximation of this library and returns a value indicating the distance between sets of points.

We can see how to apply fastdtw and polyline against Strava data in the following example:

[source,python]
----

import os
import polyline
import requests
from fastdtw import fastdtw

token = os.environ["TOKEN"]
headers = {'Authorization': "Bearer {0}".format(token)}

def find_points(activity_id):
    r = requests.get("https://www.strava.com/api/v3/activities/{0}".format(activity_id), headers=headers)
    response = r.json()
    line = response["map"]["polyline"]
    return polyline.decode(line)
----

Now let's try it out on two runs, https://www.strava.com/activities/1361109741[1361109741] and https://www.strava.com/activities/1346460542[1346460542]:

[source,python]
----

from scipy.spatial.distance import euclidean

activity1_id = 1361109741
activity2_id = 1346460542

distance, path = fastdtw(find_points(activity1_id), find_points(activity2_id), dist=euclidean)

>>> print(distance)
2.91985018100644
----

These two runs are both near my house so the value is small. Let's change the second route to be https://www.strava.com/activities/1246017379[from my trip to New York]:

[source,python]
----

activity1_id = 1361109741
activity2_id = 1246017379

distance, path = fastdtw(find_points(activity1_id), find_points(activity2_id), dist=euclidean)

>>> print(distance)
29383.492965394034
----

Much bigger!

I'm not really interested in the actual value returned but I am interested in the relative values. I'm building a little application to generate routes that I should run and I want it to come up with a routes that are different to recent ones that I've run. This score can now form part of the criteria.
Asciidoc to Asciidoc: Exploding includes
Date: 2018-01-23 21:11:49

One of my favourite features in http://asciidoctor.org/docs/asciidoc-syntax-quick-reference/#include-files[AsciiDoc] is the ability to include other files, but when using lots of includes is that it becomes difficult to read the whole document unless you convert it to one of the supported backends.

[source,bash]
----

$ asciidoctor --help
Usage: asciidoctor [OPTION]... FILE...
Translate the AsciiDoc source FILE or FILE(s) into the backend output format (e.g., HTML 5, DocBook 4.5, etc.)
By default, the output is written to a file with the basename of the source file and the appropriate extension.
Example: asciidoctor -b html5 source.asciidoc

    -b, --backend BACKEND            set output format backend: [html5, xhtml5, docbook5, docbook45, manpage] (default: html5)
                                     additional backends are supported via extensions (e.g., pdf, latex)
----

I don't want to have to convert my code to one of these formats each time - I want to convert asciidoc to asciidoc!

For example, given the following files:

+++<cite>+++mydoc.adoc+++</cite>+++

[source,text]
----

= My Blog example

== Heading 1

Some awesome text

== Heading 2

\include::blog_include.adoc[]
----

.blog_include.adoc
[source,text]
----

Some included text
----

I want to generate another asciidoc file where the contents of the include file are exploded and displayed inline.

After a lot of searching I came across https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/scripts/asciidoc-coalescer.rb[an excellent script] written by Dan Allen and put it in a file called +++<cite>+++adoc.rb+++</cite>+++. We can then call it like this:

[source,bash]
----

$ ruby adoc.rb mydoc.adoc
= My Blog example

== Heading 1

Some awesome text

== Heading 2

Some included text
----

Problem solved!

In my case I actually wanted to explode HTTP includes so I needed to pass the +++<cite>+++-a allow-uri-read+++</cite>+++ flag to the script:

[source,bash]
----

$ ruby adoc.rb mydoc.adoc -a allow-uri-read
----

And now I can generate asciidoc files until my heart's content.

</p>
Neo4j 3.4: Formatting instances of the Duration and Datetime date types
Date: 2018-06-03 04:08:21
Learn how to format the Duration and Datetime types introduced in Neo4j 3.4

In my https://markhneedham.com/blog/2018/06/02/neo4j-3.4-comparing-durations/[last blog post^] I showed how to compare instances of Neo4j's https://neo4j.com/docs/developer-manual/current/cypher/syntax/temporal/#cypher-temporal-durations[Duration data type^], and in the middle of the post I realised that I needed to use the https://neo4j-contrib.github.io/neo4j-apoc-procedures/[APOC library^] to return the value in the format I wanted.
This was the solution I ended up with:

[source, cypher]
----
WITH duration({seconds: 100}) AS duration
RETURN apoc.text.lpad(toString(duration.minutes), 2, "0") + ":" +
       apoc.text.lpad(toString(duration.secondsOfMinute), 2, "0")
----

If we run that query this is the output:

[source, text]
----
╒═══════╕
│"value"│
╞═══════╡
│"01:40"│
└───────┘
----

It works but it's not very nice so I wanted to see if I could come up with something better.

== `apoc.text.format`

My first attempt used the `apoc.text.format` function, which allows us to https://github.com/neo4j-contrib/neo4j-apoc-procedures/issues/269[format Strings in sprintf format^].
This is how we can format the duration using that function:

[source, cypher]
----
WITH duration({seconds: 100}) AS duration
RETURN apoc.text.format("%02d:%02d", [duration.minutes, duration.secondsOfMinute]) AS value
----

This is better than our first version but what if we want to include the number of hours as well?
We'd have to do this:

[source, cypher]
----
WITH duration({minutes: 100, seconds: 30}) AS duration
RETURN apoc.text.format("%02d:%02d:%02d", [duration.hours, duration.minutesOfHour, duration.secondsOfMinute]) AS value
----

If we execute that query we'll see this output:

[source, text]
----
╒══════════╕
│"value"   │
╞══════════╡
│"01:40:30"│
└──────────┘
----

Not bad.
At this point I remembered that APOC has lots of https://neo4j-contrib.github.io/neo4j-apoc-procedures/#_date_and_time_conversions[date formatting functions^] and I wondered if I could use one of them to make life even easier.

== `apoc.date.format`

Yes we can!
If we can convert our duration into milliseconds we can use `apoc.date.format` to solve the problem.

The following code does the trick:

[source, cypher]
----
WITH duration({minutes: 100, seconds: 30}) AS duration
RETURN apoc.date.format(duration.milliseconds, 'ms', 'HH:mm:ss') AS value
----

== Formatting dates

We can use the same approach to format dates as well.
We can extract the timestamp of a Datetime from the  `epochSeconds` property.
The following query formats the current Datetime:

[source, cypher]
----
RETURN apoc.date.format(datetime().epochSeconds, 's', 'MMM d yyyy') AS value
----

[source, text]
----
╒════════════╕
│"value"     │
╞════════════╡
│"Jun 3 2018"│
└────────────┘
----

If we want to extract the timestamp in milliseconds we can do that as well:

[source, cypher]
----
RETURN apoc.date.format(datetime().epochMillis, 'ms', 'MMM d yyyy') AS value
----
Pandas: Create matplotlib plot with x-axis label not index
Date: 2018-12-21 16:57:00
Learn how to set the x-axis label of a matplotxlib plot

I've been using matplotlib a bit recently, and wanted to share a lesson I learnt about choosing the label of the x-axis.
Let's first import the libraries we'll use in this post:

[source, python]
----
import pandas as pd
import matplotlib.pyplot as plt
----

And now we'll create a DataFrame of values that we want to chart:

[source,python]
----
df = pd.DataFrame({
    "name": ["Mark", "Arya", "Praveena"],
    "age": [34, 1, 31]
})
df
----

This is what our DataFrame looks like:

[source, text]
----
       name  age
0      Mark   34
1      Arya   31
2  Praveena    1
----

[source, python]
----
df.plot.bar()
plt.tight_layout()
plt.show()
----

If we run that code we'll see this chart:

image::{{<siteurl>}}/uploads/2018/12/start.svg[]

The chart itself looks fine, but the labels of the values on the x-axis are a bit weird.
They're 1, 2, and 3, whereas we want them to use the values in the `name` column of our DataFrame.

I was a bit confused at first, but eventually realised that they were the index values of our rows.
We can see that by executing the following code:

[source, python]
----
>>> df.index.values
array([0, 1, 2])
----

There are a couple of ways that we can fix our chart.
The first is to use the `name` column as our index, an approach I learnt from https://www.dataquest.io/blog/adding-axis-labels-to-plots-with-pandas-and-matplotlib/[Josh Devlin's blog post].
We can reset the index by running the following code:

[source, python]
----
df.set_index("name",drop=True,inplace=True)
----

Let's check the index values:

[source, python]
----
>>> df.index.values
array(['Mark', 'Arya', 'Praveena'], dtype=object)
----

Ah, much better!
Now we can plot our chart again:

[source, pythong]
----
df.plot.bar()
plt.tight_layout()
plt.show()
----

If we run that code we'll see this chart:

image::{{<siteurl>}}/uploads/2018/12/next.svg[]

That's much better!

We can also achieve the same outcome by specifying the `x` parameter when we call the `bar` function:

[source, python]
----
df = pd.DataFrame({
    "name": ["Mark", "Arya", "Praveena"],
    "age": [34, 1, 31]
})
df.plot.bar(x="name")
plt.tight_layout()
plt.show()
----
PySpark: Creating DataFrame with one column - TypeError: Can not infer schema for type: <type 'int'>
Date: 2018-12-09 10:25:00
Learn how to create a PySpark DataFrame with one column.

I've been playing with PySpark recently, and wanted to create a DataFrame containing only one column.
I tried to do this by writing the following code:

[source, python]
----
spark.createDataFrame([(1)], ["count"])
----

If we run that code we'll get the following error message:

[source, bash]
----
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/markhneedham/projects/graph-algorithms/spark-2.4.0-bin-hadoop2.7/python/pyspark/sql/session.py", line 748, in createDataFrame
    rdd, schema = self._createFromLocal(map(prepare, data), schema)
  File "/home/markhneedham/projects/graph-algorithms/spark-2.4.0-bin-hadoop2.7/python/pyspark/sql/session.py", line 416, in _createFromLocal
    struct = self._inferSchemaFromList(data, names=schema)
  File "/home/markhneedham/projects/graph-algorithms/spark-2.4.0-bin-hadoop2.7/python/pyspark/sql/session.py", line 348, in _inferSchemaFromList
    schema = reduce(_merge_type, (_infer_schema(row, names) for row in data))
  File "/home/markhneedham/projects/graph-algorithms/spark-2.4.0-bin-hadoop2.7/python/pyspark/sql/session.py", line 348, in <genexpr>
    schema = reduce(_merge_type, (_infer_schema(row, names) for row in data))
  File "/home/markhneedham/projects/graph-algorithms/spark-2.4.0-bin-hadoop2.7/python/pyspark/sql/types.py", line 1062, in _infer_schema
    raise TypeError("Can not infer schema for type: %s" % type(row))
TypeError: Can not infer schema for type: <type 'int'>
----

The problem we have is that `createDataFrame` http://sqlblog.com/blogs/jamie_thomson/archive/2016/12/12/creating-a-spark-dataframe-containing-only-one-column.aspx[expects a tuple of values^], and we've given it an integer.
Luckily we can fix this reasonably easily by passing in a single item tuple:

[source, python]
----
spark.createDataFrame([(1,)], ["count"])
----

If we run that code we'll get the expected DataFrame:

[options="header"]
|=======
|count
|1
|=======
Python: Pandas - DataFrame plotting ignoring figure
Date: 2018-12-25 21:09:00
Learn how to plot a Pandas DataFrame onto a custom sized chart.

In my continued use of matplotlib I wanted to change the size of the chart I was plotting and struggled a bit to start with.
We'll use the same DataFrame as before:

[source,python]
----
df = pd.DataFrame({
    "name": ["Mark", "Arya", "Praveena"],
    "age": [34, 1, 31]
})
df
----

In my last blog post I showed how we can create a bar chart by executing the following code:

[source,python]
----
df.plot.bar(x="name")
plt.tight_layout()
plt.show()
plt.close()
----

But how do we make it bigger?
We can control this https://stackoverflow.com/questions/332289/how-do-you-change-the-size-of-figures-drawn-with-matplotlib[by using^] the https://matplotlib.org/api/_as_gen/matplotlib.pyplot.figure.html#matplotlib.pyplot.figure[`figure`^] function.

I gave this a try:

[source,python]
----
plt.figure(figsize=(20,10))
df.plot.bar(x="name")
plt.tight_layout()
plt.show()
plt.close()
----

If we run that we'll see this output:

image::{{<siteurl>}}/uploads/2018/12/Selection_093.png[]

Hmmmm...we now have two figures, and the bigger one is completely blank!
That's not quite what we expected.

I came across a https://stackoverflow.com/questions/42215252/inconsistency-when-setting-figure-size-using-pandas-plot-method/42216385#42216385[really thorough StackOverflow post^] which explained a variety of ways to solve the problem.
The first way is to specify the `figsize` parameter when we call the `bar` function:

[source,python]
----
df.plot.bar(x="name", figsize=(20,10))
plt.tight_layout()
plt.show()
plt.close()
----

If we execute that code we'll now have our big chart:

image::{{<siteurl>}}/uploads/2018/12/big.svg[]

There are another couple of ways we can achieve this as well.
The `plot` function takes in a `ax` parameter, to which we can pass an existing Axes.

The `gca` function on our plot returns the current Axes instance or creates a new one:

[source,python]
----
plt.figure(figsize=(20,10))
df.plot.bar(x="name", ax=plt.gca())
plt.tight_layout()
plt.show()
plt.close()
----

Or rather than using the `gca` function on `plt`, we can capture the axes from the `figure` function and pass it in directly:

[source,python]
----
fig = plt.figure(figsize=(20,10))
df.plot.bar(x="name", ax=fig.gca())
plt.tight_layout()
plt.show()
plt.close()
----
TWU: Session Design - Measurable goals
Date: 2011-03-04 02:49:59

We've been spending our time recently preparing the sessions for the next ThoughtWorks University batch and one thing http://www.learninggeneralist.com/[Sumeet] has encouraged us to do is ensure that we have a *measurable goal for each session*.

In our case that means that we need to design our sessions with the intention of the grads being able to do something rather than understand something after the session.

It's very difficult to measure whether someone understands something and from what I've noticed having a goal of someone understanding something can encourage you to put in more than is strictly necessary.

Sumeet pointed us to Cathy Moore's http://blog.cathy-moore.com/2008/05/be-an-elearning-action-hero/[action mapping] which talks about the need to create some sort of action from the learning opportunity that we're providing.

She also talks about the need to avoid http://www.markhneedham.com/blog/2010/10/31/distributed-agile-context/[brain dumping] and I've noticed that I have a tendency to do this if my goal is for someone to 'understand topic x'.

I've found it quite difficult to work out how much information and context we should be providing but our current thinking is that providing just enough to allow the grads to work on http://www.sukrupa.org/[their project] while encouraging them to seek out more information when they need it will be the most effective approach.

On a somewhat related note, after a recommendation by my colleague http://jimbarritt.com/non-random/[Jim Barritt] I've been reading through Gerald Weinburg's 'http://www.amazon.com/Becoming-a-Technical-Leader-ebook/dp/B004J4VV3I/ref=sr_1_2?ie=UTF8&m=AG56TWVU5XWC2&s=books&qid=1299125329&sr=8-2[Becoming a Technical Leader]' and I quite like Weinburg's advice around this:

____
When I write a book or an essay or a novel, or give a workshop, I measure my success by one simple test: _When I'm finished with them, do they care less about the subject?_ If the answer if yes, I've failed.
____

I'm not sure how measurable this one is but it certainly encourages you not to make the session boring!
Retrospectives: Mini Group Discussions
Date: 2011-03-20 18:36:42

One of the approaches that I like the best in retrospectives is when the facilitator splits the team into smaller groups during the brainstorming part of the retrospective.

I decided to try this out in a retrospective we ran after one week of ThoughtWorks University, using http://www.thekua.com/rant/2006/03/the-retrospective-starfish/[The Retrospective Starfish] to provide a framework in which people could frame their thoughts.

Usually what I've seen happen in these mini groups is that everyone will write down their own ideas on stickies and then discuss them as a group but still put up all the stickies even if the group didn't agree with everything.

http://twitter.com/frankmt[Frankie] observed that in this retrospective that hadn't really happened and the mini groups had been reaching a consensus on their ideas before one or two people went and put them up on the wall.

I think this partly came down to my failure to explain the intent of the exercise clearly i.e. *it is a brainstorming exercise rather than an analytical one*.

An interesting side effect was that when people came and looked at the wall to see what the other groups had come up with they noticed quite a few things that they agreed with but hadn't come up with in their group.

I think next time I'll try and explain the exercise a bit more clearly to help keep people in the mindset of coming up with ideas.

It's difficult to say if any important points were missed by people analysing so early but there was certainly less duplication of ideas than normal!
Confirmation Bias and Loss of Autonomy
Date: 2011-03-20 18:08:35

I've mentioned confirmation bias http://www.markhneedham.com/blog/2010/07/22/the-prepared-mind-vs-having-context-when-learning-new-ideas/[in] http://www.markhneedham.com/blog/2009/08/12/zen-mind-beginners-mind-book-review/[a few] http://www.markhneedham.com/blog/2009/12/24/debug-it-book-review/[of my previous blog posts] but I hadn't realised quite how widespread it can be in organisations until quite recently.

_Confirmation bias_

____
[A] tendency for people to favor information that confirms their preconceptions or hypotheses regardless of whether the information is true. As a result, people gather evidence and recall information from memory selectively, and interpret it in a biased way. The biases appear in particular for emotionally significant issues and for established beliefs.
____

I've noticed that it is particularly prevalent in organisations when people feel that they are being told what to do i.e. people feel they don't have autonomy.

http://www.youtube.com/watch?v=u6XAPnuFjJc[Dan Pink] talks about autonomy as being one of the 3 things that lead to better performance and personal satisfaction.

_Autonomy_

____
immunity from arbitrary exercise of authority
____

One example could be if an organisation were to force a certain way of working on their employees without getting their input on it beforehand.

It's highly likely the employees will start looking for reasons why the new way of working is worse than what they have.

Even if the management can very logically explain why it's better they're unlikely to make much headway because *people will only see information which confirms their view that the old way is better*.

http://twitter.com/zaynilee[Zaynab] pointed out that a useful technique to avoid confirmation bias is to look at what you were trying to achieve in old way and see how you would do that with the new way. That way you lose nothing!

I think this approach only really works once the http://stevenmsmith.com/ar-satir-change-model/[emotions of the change] have calmed down a bit though.

Equally the management may be prone to confirmation bias since they have bought into the fact that the new way is better and may dismiss potentially valid feedback because it doesn't fit with their belief system of how things should be.

I feel like confirmation bias is quite a natural human reaction to a situation like this so it's useful to be aware of it if you're having something forced onto you but also if you're in a position of instigating some sort of organisational change.
ThoughtWorks University: Pulling the 'pearls'
Date: 2011-03-29 18:32:14

I recently wrote about the http://www.markhneedham.com/blog/2011/03/29/thoughtworks-university-coding-dojo-style/[coding dojo style week that we ran at ThoughtWorks University last week] and I briefly mentioned that we used break out sessions to cover topics ('the pearls') that people didn't totally understand.

To describe that in more detail what we did to start with was write the name of each of the 90/180 minute sessions on a card and put it on the wall under a 'To Do' heading:+++<div align="center">+++image::{{<siteurl>}}/uploads/2011/03/twu-topics-todo.jpg[Twu topics todo,271]

image::{{<siteurl>}}/uploads/2011/03/twu-topics-done.jpg[Twu topics done,271]+++</div>+++

There were about 10 of those topics and our goal was to cut each of those topics down to a 20 minute introduction that one of the trainers could give when it seemed like a good time.

For example we did a brief introduction to 'Build and Deployment' when we first talked about continuous integration and we went through the application architecture just before we started coding our first story.

Our goal with this approach was to *encourage a pull based approach to learning over a push based one* where we  just presented on different topics without giving the grads a chance to properly see if they understood the topic.

The reason that we did such short presentations is that we didn't feel you can learn that much from a presenation so we wanted to give just enough information that the grads would be able to go and explore the topic more on their own.

To push this further we asked the grads to add any other topics they wanted us to talk about to the board so that we could cover those when we got the chance.

While we were doing these introductory presentations http://twitter.com/frankmt[Frankie] and http://twitter.com/jimbarritt[Jim] came up with the idea of asking the grads to rate themselves 1-5 on their current understanding of the topic and to also give a time limit for the presentation.

We repeated the rating afterwards to see if the introduction had actually been useful and then spent time discussing the topic further if necessary.

This approach worked pretty well for helping avoid the http://www.markhneedham.com/blog/2011/03/23/twu-brain-dumping/[brain dumping] that I wrote about last week and the rating system has helped give us an idea of which topics are considered more difficult by the group.

Hopefully within the next few weeks we'll start to see the grads running their own sessions which will http://www.markhneedham.com/blog/2009/04/21/learning-through-teaching/[take their learning even further].