markhneedham/blog huggingface dataset:
Kubernetes: Simple example of pod running
Date: 2017-10-21 10:06:55

I recently needed to create a Kubernetes pod that would 'just sit there' while I used +++<cite>+++kube cp+++</cite>+++ to copy some files to a persistent volume to which it was bound.

I started out with this naive pod spec:

+++<cite>+++pod_no_while.yaml+++</cite>+++

[source,yaml]
----

kind: Pod
apiVersion: v1
metadata:
  name: marks-dummy-pod
spec:
  containers:
    - name: marks-dummy-pod
      image: ubuntu
  restartPolicy: Never
----

Let's apply that template:

[source,bash]
----

$ kubectl apply -f pod_no_while.yaml
pod "marks-dummy-pod" created
----

And let's check if we have any running pods:

[source,bash]
----

$ kubectl get pods
No resources found, use --show-all to see completed objects.
----

We won't see anything here because unsurprisingly the pod has run to completion as there's nothing to keep it running! We can confirm that by running this command:

[source,bash]
----

$ kubectl get pods --show-all
NAME              READY     STATUS      RESTARTS   AGE
marks-dummy-pod   0/1       Completed   0          1m
----

Now let's create a pod that has an infinite bash while loop:

+++<cite>+++pod.yaml+++</cite>+++

[source,yaml]
----

kind: Pod
apiVersion: v1
metadata:
  name: marks-dummy-pod
spec:
  containers:
    - name: marks-dummy-pod
      image: ubuntu
      command: ["/bin/bash", "-ec", "while :; do echo '.'; sleep 5 ; done"]
  restartPolicy: Never
----

Let's apply that one instead:

[source,bash]
----

$ kubectl apply -f pod.yaml
The Pod "marks-dummy-pod" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)
----

Oops, we need to delete it first so let's do that:

[source,bash]
----

$ kubectl delete pod marks-dummy-pod
pod "marks-dummy-pod" deleted
----

Attempt #2:

[source,bash]
----

$ kubectl apply -f pod.yaml
pod "marks-dummy-pod" created
----

And let's checkup on our pod:

[source,bash]
----

$ kubectl get pods
NAME              READY     STATUS    RESTARTS   AGE
marks-dummy-pod   1/1       Running   0          14s
----

Looks better already. Let's check the logs

[source,bash]
----

$ kubectl logs marks-dummy-pod
.
.
----

Great! We can now +++<cite>+++kubectl cp+++</cite>+++ to our heart's content and then delete the pod aftewards.
Neo4j: Traversal query timeout
Date: 2017-10-31 21:43:17
Learn how to set a timeout in a Neo4j user defined procedure that uses the traversal API.

I've been spending some of my spare time over the last few weeks creating an application that generates running routes from https://www.ordnancesurvey.co.uk/business-and-government/products/os-open-roads.html[Open Roads] data - transformed and imported into Neo4j of course!

I've created a user defined procedure which combines several shortest path queries, but I wanted to exit any of these shortest path searches if they were taking too long. My code without a timeout looks like this:

[source,java]
----

StandardExpander orderedExpander = new OrderedByTypeExpander()
    .add( RelationshipType.withName( "CONNECTS" ), Direction.BOTH );

PathFinder<Path> shortestPathFinder = GraphAlgoFactory.shortestPath( expander, 250 );

...
----

There are several places where we could check the time elapsed, but the +++<cite>+++expand+++</cite>+++ method in the +++<cite>+++Expander+++</cite>+++ seemed like an obvious one to me. I wrote my own +++<cite>+++Expander+++</cite>+++ class which looks like this:

[source,java]
----

public class TimeConstrainedExpander implements PathExpander
{
    private final StandardExpander expander;
    private final long startTime;
    private final Clock clock;
    private int pathsExpanded = 0;
    private long timeLimitInMillis;

    public TimeConstrainedExpander( StandardExpander expander, Clock clock, long timeLimitInMillis )
    {
        this.expander = expander;
        this.clock = clock;
        this.startTime = clock.instant().toEpochMilli();
        this.timeLimitInMillis = timeLimitInMillis;
    }

    @Override
    public Iterable<Relationship> expand( Path path, BranchState state )
    {
        long timeSoFar = clock.instant().toEpochMilli() - startTime;
        if ( timeSoFar > timeLimitInMillis )
        {
            return Collections.emptyList();
        }

        return expander.expand( path, state );
    }

    @Override
    public PathExpander reverse()
    {
        return expander.reverse();
    }
}
----

The code snippet from earlier now needs to be updated to use our new class, which isn't too tricky:

[source,java]
----

StandardExpander orderedExpander = new OrderedByTypeExpander()
    .add( RelationshipType.withName( "CONNECTS" ), Direction.BOTH );

TimeConstrainedExpander expander = new TimeConstrainedExpander(orderedExpander,
    Clock.systemUTC(), 200);

PathFinder<Path> shortestPathFinder = GraphAlgoFactory.shortestPath( expander, 250 );
...
----

I'm not sure if this is the best way to achieve what I want but after failing with several other approaches at least this one actually works!
Pandas: ValueError: The truth value of a Series is ambiguous.
Date: 2017-07-26 21:41:55
In this post we look at how to work around the ValueError: The truth value of a Series is ambiguous when adding a new column to a DataFrame.

I've been playing around with Kaggle in my spare time over the last few weeks and came across an unexpected behaviour when trying to add a column to a dataframe.

First let's get Panda's into our program scope:

== Prerequisites

[source,python]
----

import pandas as pd
----

Now we'll create a data frame to play with for the duration of this post:

[source,python]
----

>>> df = pd.DataFrame({"a": [1,2,3,4,5], "b": [2,3,4,5,6]})
>>> df
   a  b
0  5  2
1  6  6
2  0  8
3  3  2
4  1  6
----

Let's say we want to create a new column which returns True if either of the numbers are odd. If not then it'll return False.

We'd expect to see a column full of True values so let's get started.

[source,python]
----

>>> divmod(df["a"], 2)[1] > 0
0     True
1    False
2     True
3    False
4     True
Name: a, dtype: bool

>>> divmod(df["b"], 2)[1] > 0
0    False
1     True
2    False
3     True
4    False
Name: b, dtype: bool
----

So far so good. Now let's combine those two calculations together and create a new column in our data frame:

[source,python]
----

>>> df["anyOdd"] = (divmod(df["a"], 2)[1] > 0) or (divmod(df["b"], 2)[1] > 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/markneedham/projects/kaggle/house-prices/a/lib/python3.6/site-packages/pandas/core/generic.py", line 953, in __nonzero__
    .format(self.__class__.__name__))
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
----

Hmmm, that was unexpected! Unfortunately Python's +++<cite>+++or+++</cite>+++ and +++<cite>+++and+++</cite>+++ statements https://stackoverflow.com/questions/36921951/truth-value-of-a-series-is-ambiguous-use-a-empty-a-bool-a-item-a-any-o[don't work very well against Panda's Series'], so instead we need to use the bitwise or (|) and and (&).

Let's update our example:

[source,python]
----

>>> df["anyOdd"] = (divmod(df["a"], 2)[1] > 0) | (divmod(df["b"], 2)[1] > 0)
>>> df
   a  b  anyOdd
0  1  2    True
1  2  3    True
2  3  4    True
3  4  5    True
4  5  6    True
----

Much better. And what about if we wanted to check if both values are odd?

[source,python]
----

>>> df["bothOdd"] = (divmod(df["a"], 2)[1] > 0) & (divmod(df["b"], 2)[1] > 0)
>>> df
   a  b  anyOdd  bothOdd
0  1  2    True    False
1  2  3    True    False
2  3  4    True    False
3  4  5    True    False
4  5  6    True    False
----

Works exactly as expected, hoorah!
Neo4j: Cypher - Create Cypher map with dynamic keys
Date: 2017-09-19 19:30:09
Learn how to create maps with dynamic keys using Neo4j's Cypher query language and the APOC procedure library.

I was recently trying to create a map in a Cypher query but wanted to have dynamic keys in that map. I started off with this query:

[source,cypher]
----

WITH "a" as dynamicKey, "b" as dynamicValue
RETURN { dynamicKey: dynamicValue } AS map


╒══════════════════╕
│"map"             │
╞══════════════════╡
│{"dynamicKey":"b"}│
└──────────────────┘
----

Not quite what we want! We want +++<cite>+++dynamicKey+++</cite>+++ to be evaluated rather than treated as a literal. As usual, https://github.com/neo4j-contrib/neo4j-apoc-procedures[APOC] comes to the rescue!

In fact APOC has several functions that will help us out here. Let's take a look at them:

[source,cypher]
----

CALL dbms.functions() yield name, description
WHERE name STARTS WITH "apoc.map.from"
RETURN name, description

╒═════════════════════╤═════════════════════════════════════════════════════╕
│"name"               │"description"                                        │
╞═════════════════════╪═════════════════════════════════════════════════════╡
│"apoc.map.fromLists" │"apoc.map.fromLists([keys],[values])"                │
├─────────────────────┼─────────────────────────────────────────────────────┤
│"apoc.map.fromNodes" │"apoc.map.fromNodes(label, property)"                │
├─────────────────────┼─────────────────────────────────────────────────────┤
│"apoc.map.fromPairs" │"apoc.map.fromPairs([[key,value],[key2,value2],...])"│
├─────────────────────┼─────────────────────────────────────────────────────┤
│"apoc.map.fromValues"│"apoc.map.fromValues([key1,value1,key2,value2,...])" │
└─────────────────────┴─────────────────────────────────────────────────────┘
----

So we can generate a map like this:

[source,cypher]
----

WITH "a" as dynamicKey, "b" as dynamicValue
RETURN apoc.map.fromValues([dynamicKey, dynamicValue]) AS map

╒═════════╕
│"map"    │
╞═════════╡
│{"a":"b"}│
└─────────┘
----

or like this:

[source,cypher]
----

WITH "a" as dynamicKey, "b" as dynamicValue
RETURN apoc.map.fromLists([dynamicKey], [dynamicValue]) AS map

╒═════════╕
│"map"    │
╞═════════╡
│{"a":"b"}│
└─────────┘
----

or even like this:

[source,cypher]
----

WITH "a" as dynamicKey, "b" as dynamicValue
RETURN apoc.map.fromPairs([[dynamicKey, dynamicValue]]) AS map

╒═════════╕
│"map"    │
╞═════════╡
│{"a":"b"}│
└─────────┘
----
Python 3: Create sparklines using matplotlib
Date: 2017-09-23 06:51:56
Learn how to create Edward Tufte style sparklines graphics using Python's matplotlib libary.

I recently wanted to create https://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0001OR[sparklines] to show how some values were changing over time. In addition, I wanted to generate them as images on the server rather than introducing a JavaScript library.

Chris Seymour's https://github.com/iiSeymour/sparkline-nb/blob/master/sparkline-nb.ipynb[excellent gist] which shows how to create sparklines inside a Pandas dataframe got me most of the way there, but I had to tweak his code a bit to get it to play nicely with Python 3.6.

This is what I ended up with:

[source,python]
----

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import base64

from io import BytesIO

def sparkline(data, figsize=(4, 0.25), **kwags):
    """
    Returns a HTML image tag containing a base64 encoded sparkline style plot
    """
    data = list(data)

    fig, ax = plt.subplots(1, 1, figsize=figsize, **kwags)
    ax.plot(data)
    for k,v in ax.spines.items():
        v.set_visible(False)
    ax.set_xticks([])
    ax.set_yticks([])

    plt.plot(len(data) - 1, data[len(data) - 1], 'r.')

    ax.fill_between(range(len(data)), data, len(data)*[min(data)], alpha=0.1)

    img = BytesIO()
    plt.savefig(img, transparent=True, bbox_inches='tight')
    img.seek(0)
    plt.close()

    return base64.b64encode(img.read()).decode("UTF-8")
----

I had to change the class used to write the image from StringIO to BytesIO and I found I needed to decode the bytes produced if I wanted it to display in a HTML page.

This is how you would call the above function:

[source,python]
----

if __name__ == "__main__":
    values = [
        [1,2,3,4,5,6,7,8,9,10],
        [7,10,12,18,2,8,10,6,7,12],
        [10,9,8,7,6,5,4,3,2,1]
    ]

    with open("/tmp/foo.html", "w") as file:
        for value in values:
            file.write('<div><img src="data:image/png;base64,{}"/></div>'.format(sparkline(value)))
----

And the HTML page looks like this:

image::{{<siteurl>}}/uploads/2017/09/2017-09-23_07-49-32.png[2017 09 23 07 49 32,160]
AWS Lambda: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory'
Date: 2017-08-03 17:24:16

I've been working on an AWS lambda job to convert a HTML page to PDF using a Python wrapper around the wkhtmltopdf library but ended up with the following error when I tried to execute it:

[source,text]
----

b'/bin/sh: ./binary/wkhtmltopdf: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory\n': Exception
Traceback (most recent call last):
File "/var/task/handler.py", line 33, in generate_certificate
wkhtmltopdf(local_html_file_name, local_pdf_file_name)
File "/var/task/lib/wkhtmltopdf.py", line 64, in wkhtmltopdf
wkhp.render()
File "/var/task/lib/wkhtmltopdf.py", line 56, in render
raise Exception(stderr)
Exception: b'/bin/sh: ./binary/wkhtmltopdf: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory\n'
----

It turns out this is the error you get if https://stackoverflow.com/questions/8328250/centos-64-bit-bad-elf-interpreter[you run a 32 bit binary on a 64 bit operating system] which is http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html[what AWS uses].

____
If you are using any native binaries in your code, make sure they are compiled in this environment. Note that only 64-bit binaries are supported on AWS Lambda.
____

I changed to the 64 bit binary and am now happily converting HTML pages to PDF.
PHP vs Python: Generating a HMAC
Date: 2017-08-02 06:09:10
In this post I show how to generate a Base 64 encoded HMAC using Python and compare it to the same code in PHP.

I've been writing a bit of code to integrate with a ClassMarker webhook, and you're required to check that an incoming request actually came from ClassMarker by https://www.classmarker.com/online-testing/api/webhooks/#verify[checking the value of a base64 hash using HMAC SHA256].

The example in the documentation is written in PHP which I haven't done for about 10 years so I had to figure out how to do the same thing in Python.

This is the PHP version:

[source,bash]
----

$ php -a
php > echo base64_encode(hash_hmac("sha256", "my data", "my_secret", true));
vyniKpNSlxu4AfTgSJImt+j+pRx7v6m+YBobfKsoGhE=
----

The Python equivalent is a https://stackoverflow.com/a/1306575[bit more code] but it's not too bad.

== Import all the libraries

[source,python]
----

import hmac
import hashlib
import base64
----

== Generate that hash

[source,bash]
----

data = "my data".encode("utf-8")
digest = hmac.new(b"my_secret", data, digestmod=hashlib.sha256).digest()

print(base64.b64encode(digest).decode())
'vyniKpNSlxu4AfTgSJImt+j+pRx7v6m+YBobfKsoGhE='
----

We're getting the same value as the PHP version so it's good times all round.
Serverless: AWS HTTP Gateway - 502 Bad Gateway
Date: 2017-08-11 16:01:50
Learn how to get up and running with a simple Python Serverless function that works with an AWS HTTP gateway.

In my continued work with Serverless and AWS Lambda I ran into a problem when trying to call a https://serverless.com/framework/docs/providers/aws/events/apigateway/[HTTP gateway].

My project looked like this:

_serverless.yaml_

[source,text]
----

service: http-gateway

frameworkVersion: ">=1.2.0 <2.0.0"

provider:
  name: aws
  runtime: python3.6
  timeout: 180

functions:
  no-op:
      name: NoOp
      handler: handler.noop
      events:
        - http: POST noOp
----

_handler.py_

[source,python]
----

def noop(event, context):
    return "hello"
----

Let's deploy to AWS:

[source,bash]
----

$ serverless  deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (179 B)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
..............
Serverless: Stack update finished...
Service Information
service: http-gateway
stage: dev
region: us-east-1
api keys:
  None
endpoints:
  POST - https://29nb5rmmd0.execute-api.us-east-1.amazonaws.com/dev/noOp
functions:
  no-op: http-gateway-dev-no-op
----

And now we'll try and call it using cURL:

[source,bash]
----

$ curl -X POST https://29nb5rmmd0.execute-api.us-east-1.amazonaws.com/dev/noOp
{"message": "Internal server error"}
----

That didn't work so well, what do the logs have to say?

[source,bash]
----

$ serverless  logs --function no-op	
START RequestId: 64ab69b0-7d8f-11e7-9db5-13b228cd4cb6 Version: $LATEST
END RequestId: 64ab69b0-7d8f-11e7-9db5-13b228cd4cb6
REPORT RequestId: 64ab69b0-7d8f-11e7-9db5-13b228cd4cb6	Duration: 0.27 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 21 MB
----

So the function is completely fine. It turns out I'm not very good at reading the manual and should have been returning https://medium.com/@jconning/tutorial-aws-lambda-with-api-gateway-36a8513ec8e3[a map instead of a string]:

____
API Gateway expects to see a json map with keys "`body`", "`headers`", and "`statusCode`".
____

Let's update our handler function and re-deploy.

[source,python]
----

def noop(event, context):
    return {
        "body": "hello",
        "headers": {},
        "statusCode": 200
        }
----

Now we're ready to try the endpoint again:

[source,bash]
----

$ curl -X POST https://29nb5rmmd0.execute-api.us-east-1.amazonaws.com/dev/noOp
hello
----

Much better!
Neo4j: Cypher - Rounding of floating point numbers/BigDecimals
Date: 2017-08-13 07:23:46
Learn how to deal with floating point arithmetic and rounding errors when working with Neo4j's Cypher query language. APOC comes to the rescue.

I was doing some data cleaning a few days ago and wanting to multiply a value by 1 million. My https://neo4j.com/developer/cypher-query-language/[Cypher] code to do this looked like this:

[source,cypher]
----

with "8.37" as rawNumeric
RETURN toFloat(rawNumeric) * 1000000 AS numeric

╒═════════════════╕
│"numeric"        │
╞═════════════════╡
│8369999.999999999│
└─────────────────┘
----

Unfortunately that suffers from the classic rounding error when working with floating point numbers. I couldn't figure out a way to solve it using pure Cypher, but there tends to be an https://neo4j-contrib.github.io/neo4j-apoc-procedures/[APOC] function to solve every problem and https://neo4j-contrib.github.io/neo4j-apoc-procedures/index32.html#_handle_biginteger_and_bigdecimal[this was no exception].

I'm using Neo4j 3.2.3 so I downloaded the corresponding APOC jar and put it in a plugins directory:

[source,bash]
----

$ ls -lh plugins/
total 3664
-rw-r--r--@ 1 markneedham  staff   1.8M  9 Aug 09:14 apoc-3.2.0.4-all.jar
----

I'm using Docker so I needed to tell that where my plugins folder lives:

[source,bash]
----

$ docker run -v $PWD/plugins:/plugins \
    -p 7474:7474 \
    -p 7687:7687 \
    -e NEO4J_AUTH="none" \
    neo4j:3.2.3
----

Now we're reading to try out our new function:

[source,cypher]
----

with "8.37" as rawNumeric
RETURN apoc.number.exact.mul(rawNumeric,"1000000") AS apocConversion

╒════════════════╕
│"apocConversion"│
╞════════════════╡
│"8370000.00"    │
└────────────────┘
----

That almost does what we want, but the result is a string rather than numeric value. It's not too difficult to fix though:

[source,cypher]
----

with "8.37" as rawNumeric
RETURN toFloat(apoc.number.exact.mul(rawNumeric,"1000000")) AS apocConversion

╒════════════════╕
│"apocConversion"│
╞════════════════╡
│8370000         │
└────────────────┘
----

That's more like it!
Go: Multi-threaded writing to a CSV file
Date: 2017-01-31 05:57:11
Writing to a CSV from multiple threads in the Go programming language.

As part of a Go script I've been working on I wanted to write to a CSV file from multiple Go routines, but realised that the built in CSV Writer isn't thread safe.

My first attempt at writing to the CSV file looked like this:

[source,go]
----

package main


import (
	"encoding/csv"
	"os"
	"log"
	"strconv"
)

func main() {

	csvFile, err := os.Create("/tmp/foo.csv")
	if err != nil {
		log.Panic(err)
	}

	w := csv.NewWriter(csvFile)
	w.Write([]string{"id1","id2","id3"})

	count := 100
	done := make(chan bool, count)

	for i := 0; i < count; i++ {
		go func(i int) {
			w.Write([]string {strconv.Itoa(i), strconv.Itoa(i), strconv.Itoa(i)})
			done <- true
		}(i)
	}

	for i:=0; i < count; i++ {
		<- done
	}
	w.Flush()
}
----

This script should output the numbers from 0-99 three times on each line. Some rows in the file are written correctly, but as we can see below, some aren't:

[source,text]
----

40,40,40
37,37,37
38,38,38
18,18,39
^@,39,39
...
67,67,70,^@70,70
65,65,65
73,73,73
66,66,66
72,72,72
75,74,75,74,75
74
7779^@,79,77
...
----

One way that we can make our script safe is to use a mutex whenever we're calling any methods on the CSV writer. I wrote the following code to do this:

[source,go]
----

type CsvWriter struct {
	mutex *sync.Mutex
	csvWriter *csv.Writer
}

func NewCsvWriter(fileName string) (*CsvWriter, error) {
	csvFile, err := os.Create(fileName)
	if err != nil {
		return nil, err
	}
	w := csv.NewWriter(csvFile)
	return &CsvWriter{csvWriter:w, mutex: &sync.Mutex{}}, nil
}

func (w *CsvWriter) Write(row []string) {
	w.mutex.Lock()
	w.csvWriter.Write(row)
	w.mutex.Unlock()
}

func (w *CsvWriter) Flush() {
	w.mutex.Lock()
	w.csvWriter.Flush()
	w.mutex.Unlock()
}
----

We create a mutex when +++<cite>+++NewCsvWriter+++</cite>+++ instantiates +++<cite>+++CsvWriter+++</cite>+++ and then use it in the +++<cite>+++Write+++</cite>+++ and +++<cite>+++Flush+++</cite>+++ functions so that only one go routine at a time can access the underlying +++<cite>+++CsvWriter+++</cite>+++. We then tweak the initial script to call this class instead of calling CsvWriter directly:

[source,go]
----

func main() {
	w, err := NewCsvWriter("/tmp/foo-safe.csv")
	if err != nil {
		log.Panic(err)
	}

	w.Write([]string{"id1","id2","id3"})

	count := 100
	done := make(chan bool, count)

	for i := 0; i < count; i++ {
		go func(i int) {
			w.Write([]string {strconv.Itoa(i), strconv.Itoa(i), strconv.Itoa(i)})
			done <- true
		}(i)
	}

	for i:=0; i < count; i++ {
		<- done
	}
	w.Flush()
}
----

And now if we inspect the CSV file all lines have been written successfully:

[source,text]
----

...
25,25,25
13,13,13
29,29,29
32,32,32
26,26,26
30,30,30
27,27,27
31,31,31
28,28,28
34,34,34
35,35,35
33,33,33
37,37,37
36,36,36
...
----

That's all for now. If you have any suggestions for a better way to do this do let me know in the comments or on twitter - I'm https://twitter.com/markhneedham[@markhneedham]