markhneedham/blog huggingface dataset:
Python 3: TypeError: Object of type 'dict_values' is not JSON serializable
Date: 2017-03-19 16:40:03

I've recently upgraded to Python 3 (I know, took me a while!) and realised that one of my scripts that writes JSON to a file no longer works!

This is a simplified version of what I'm doing:

[source,python]
----

>>> import json
>>> x = {"mark": {"name": "Mark"}, "michael": {"name": "Michael"}  }
>>> json.dumps(x.values())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'dict_values' is not JSON serializable
----

Python 2.7 would be perfectly happy:

[source,python]
----

>>> json.dumps(x.values())
'[{"name": "Michael"}, {"name": "Mark"}]'
----

The difference is in the results returned by the +++<cite>+++values+++</cite>+++ method:

[source,python]
----

# Python 2.7.10
>>> x.values()
[{'name': 'Michael'}, {'name': 'Mark'}]

# Python 3.6.0
>>> x.values()
dict_values([{'name': 'Mark'}, {'name': 'Michael'}])
>>>
----

Python 3 no longer returns an array, instead we have a +++<cite>+++dict_values+++</cite>+++ wrapper around the data.

Luckily this is easy to resolve - we just need to http://stackoverflow.com/questions/16228248/python-simplest-way-to-get-list-of-values-from-dict[wrap the call to +++<cite>+++values+++</cite>+++ with a call to +++<cite>+++list+++</cite>+++]:

[source,python]
----

>>> json.dumps(list(x.values()))
'[{"name": "Mark"}, {"name": "Michael"}]'
----

This versions works with Python 2.7 as well so if I accidentally run the script with an old version the world isn't going to explode.
Neo4j: apoc.date.parse - java.lang.IllegalArgumentException: Illegal pattern character 'T' / java.text.ParseException: Unparseable date: "2012-11-12T08:46:15Z"
Date: 2017-03-06 20:52:01

I often find myself wanting to convert date strings into Unix timestamps using https://github.com/neo4j-contrib/neo4j-apoc-procedures[Neo4j's APOC library] and unfortunately some sources don't use the format that https://neo4j-contrib.github.io/neo4j-apoc-procedures/#_date_time_support[+++<cite>+++apoc.date.parse+++</cite>+++] expects.

e.g.

[source,cypher]
----

return apoc.date.parse("2012-11-12T08:46:15Z",'s')
AS ts

Failed to invoke function `apoc.date.parse`:
Caused by: java.lang.IllegalArgumentException: java.text.ParseException: Unparseable date: "2012-11-12T08:46:15Z"
----

We need to define the format explicitly so the https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html[SimpleDataFormat] documentation comes in handy. I tried the following:

[source,cypher]
----

return apoc.date.parse("2012-11-12T08:46:15Z",'s',"yyyy-MM-ddTHH:mm:ssZ")
AS ts

Failed to invoke function `apoc.date.parse`:
Caused by: java.lang.IllegalArgumentException: Illegal pattern character 'T'
----

Hmmm, we need to quote the 'T' character - we can't just include it in the pattern. Let's try again:

[source,cypher]
----

return  apoc.date.parse("2012-11-12T08:46:15Z",'s',"yyyy-MM-dd'T'HH:mm:ssZ")
AS ts

Failed to invoke function `apoc.date.parse`:
Caused by: java.lang.IllegalArgumentException: java.text.ParseException: Unparseable date: "2012-11-12T08:46:15Z"
----

The problem now is that we haven't quoted the 'Z' but the error doesn't indicate that - not sure why!

We can either quote the 'Z':

[source,cypher]
----

return  apoc.date.parse("2012-11-12T08:46:15Z",'s',"yyyy-MM-dd'T'HH:mm:ss'Z'")
AS ts

╒══════════╕
│"ts"      │
╞══════════╡
│1352709975│
└──────────┘
----

Or we can match the timezone using 'XXX':

[source,cypher]
----

return  apoc.date.parse("2012-11-12T08:46:15Z",'s',"yyyy-MM-dd'T'HH:mm:ssXXX")
AS ts

╒══════════╕
│"ts"      │
╞══════════╡
│1352709975│
└──────────┘
----
AWS Lambda: Programatically create a Python 'Hello World' function
Date: 2017-04-02 22:11:47
Learn how to programatically create a Python AWS Lambda function and all the surrounding config.

I've been playing around with http://docs.aws.amazon.com/lambda/latest/dg/welcome.html[AWS Lambda] over the last couple of weeks and I wanted to automate the creation of these functions and all their surrounding config.

Let's say we have the following +++<cite>+++Hello World+++</cite>+++ function: ~~~python def lambda_handler(event, context): print("Hello world") ~~~

To upload it to AWS we need to put it inside a zip file so let's do that: ~~~bash $ zip HelloWorld.zip HelloWorld.py ~~~ ~~~bash $ unzip -l HelloWorld.zip Archive: HelloWorld.zip Length Date Time Name -------- ---- ---- ---- 61 04-02-17 22:04 HelloWorld.py -------- ------- 61 1 file ~~~

Now we're ready to write a script to create our AWS lambda function. ~~~python import boto3 lambda_client = boto3.client('lambda') fn_name = "HelloWorld" fn_role = 'arn:aws:iam::[your-aws-id]:role/lambda_basic_execution' lambda_client.create_function( FunctionName=fn_name, Runtime='python2.7', Role=fn_role, Handler="\{0}.lambda_handler".format(fn_name), Code={'ZipFile': open("\{0}.zip".format(fn_name), 'rb').read(), }, ) ~~~

+++<cite>+++[your-aws-id]+++</cite>+++ needs to be replaced with the identifier of our AWS account. We can find that out be running http://stackoverflow.com/questions/33791069/quick-way-to-get-aws-account-number-from-the-cli-tools[the following command] against the AWS CLI: ~~~bash $ aws ec2 describe-security-groups --query 'SecurityGroups[0].OwnerId' --output text 123456789012 ~~~

Now we can create our function: ~~~bash $ python CreateHelloWorld.py ~~~

image::{{<siteurl>}}/uploads/2017/04/2017-04-02_23-07-38.png[2017 04 02 23 07 38,262]

And if we test the function we'll get the expected output:

image::{{<siteurl>}}/uploads/2017/04/2017-04-02_23-02-59.png[2017 04 02 23 02 59,376]
Python: Flask - Generating a static HTML page
Date: 2017-04-27 20:59:56
Learn how to generate a static HTML page using Python's Flask library.

Whenever I need to quickly spin up a web application Python's http://flask.pocoo.org[Flask] library is my go to tool but I recently found myself wanting to generate a static HTML to upload to S3 and wondered if I could use it for that as well.

It's actually not too tricky. If we're in http://stackoverflow.com/questions/31830663/how-to-render-template-in-flask-without-using-request-context[the scope of the app context] then we have access to the template rendering that we'd normally use when serving the response to a web request.

The following code will generate a HTML file based on a template file</a> +++<cite>+++templates/blog.html+++<cite>+++: </p> ~~~python from flask import render_template import flask app = flask.Flask('my app') if __name__ == "__main__": with app.app_context(): rendered = render_template('blog.html', \ title = "My Generated Page", \ people = [{"name": "Mark"}, {"name": "Michael"}]) print(rendered) ~~~

+++<cite>+++templates/blog.html+++</cite>+++ ~~~html <!doctype html> +++<html>++++++<head>++++++<title>+++{{ title }}+++</title>++++++</head>++++++<body>+++* {{ person.name }}+++</body>++++++</html>+++ ~~~

If we execute the Python script it will generate the following HTML: ~~~bash $ python blog.py <!doctype html> +++<html>++++++<head>++++++<title>+++My Generated Page+++</title>++++++</head>++++++<body>+++= My Generated Page

* Mark
* Michael+++</body>++++++</html>+++ ~~~

And we can finish off by redirecting that output into a file: ~~~bash $ python blog.py > generated_blog.html ~~~

We could also write to the file from Python but this seems just as easy! +++</cite>++++++</cite>+++
GraphQL-Europe: A trip to Berlin
Date: 2017-05-27 11:31:08

Last weekend my colleagues https://twitter.com/lyonwj?lang=en[Will], https://twitter.com/mesirii?lang=en[Michael], https://twitter.com/oskarhane?lang=en[Oskar], and I went to Berlin to spend Sunday at the https://graphql-europe.org/[GraphQL Europe] conference in Berlin.

image::{{<siteurl>}}/uploads/2017/05/IMG_20170521_084449.jpg[IMG 20170521 084449]

Neo4j sponsored the conference as we've been experimenting with building a GraphQL to Neo4j integration and wanted to get some feedback from the community as well as learn what's going on in GraphQL land.

Will and Michael have https://neo4j.com/blog/graphql-neo4j-graph-database-integration/[written about their experience] where they talk more about the hackathon we hosted so I'll cover it more from a personal perspective.

The first thing that stood out for me was how busy it was - I knew GraphQL was pretty hipster but I wasn't expecting there to be ~ 300 attendees.

The venue was amazing - the nHow Hotel is located right next to the Spree River so there were great views to be had during the breaks. It also helped that it was really sunny for the whole day!

image::{{<siteurl>}}/uploads/2017/05/IMG_20170521_103636.jpg[IMG 20170521 103636]

I spent most of the day hanging out at the Neo4j booth which was good fun - several people pointed out that an integration between Neo4j and GraphQL made a lot of sense given that *GraphQL talks about the application graph and Neo4j graphs in general*.

I managed to attend a few of the talks, including one by Brooks Swinnerton from GitHub who announced that they'd be https://developer.github.com/v4/[moving to GraphQL for v4 of their API].

The most interesting part of the talk for me was when Brooks said they'd directed requests for their REST API to the GraphQL one behind the scenes for a while now to check that it could handle the load.

____
GitHub is moving to GraphQL for v4 of our API because it offers significantly more flexibility for our integrators. The ability to define precisely the data you want--and only the data you want--is a powerful advantage over the REST API v3 endpoints.
____

I think twitter may be doing something similar based on this tweet by https://twitter.com/tgvashworth[Tom Ashworth]:

++++
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Heh. Twitter GraphQL is quietly serving more than 40 million queries per day. Tiny at Twitter scale but not a bad start.</p>&mdash; Tom Ashworth (@tgvashworth) <a href="https://twitter.com/tgvashworth/status/862049341472522240?ref_src=twsrc%5Etfw">May 9, 2017</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
++++

From what I could tell the early pick up of GraphQL seems to be from the front end of applications - several of the attendees had attended https://www.react-europe.org/[ReactEurope] a couple of days earlier - but micro services were mentioned in a few of the talks and it was suggested that GraphQL works well in this world as well.

It was a fun day out so thanks to the folks at https://www.graph.cool/[Graphcool] for organising!
PostgreSQL: ERROR:  argument of WHERE must not return a set
Date: 2017-05-01 20:42:07

In my last post I showed how to http://www.markhneedham.com/blog/2017/05/01/loading-and-analysing-strava-runs-using-postgresql-json-data-type/[load and query data from the Strava API in PostgreSQL] and after executing some simple queries my next task was to query more complex part of the JSON structure.

image::{{<siteurl>}}/uploads/2017/05/2017-05-01_21-22-55.png[2017 05 01 21 22 55,187]

Strava allows users to create https://support.strava.com/hc/en-us/articles/216918167-What-are-Segments-[segments], which are edited portions of road or trail where athletes can compete for time.

I wanted to write a query to find all the times that I'd run a particular segment. e.g. the https://www.strava.com/segments/6818475[Akerman Road] segment covers a road running North to South in Kennington/Stockwell in South London.

This segment has the id +++<cite>+++'6818475'+++</cite>+++ so we'll need to look inside +++<cite>+++segment_efforts+++</cite>+++ and then compare the value +++<cite>+++segment.id+++</cite>+++ against this id.

I initially wrote this query to try and find the times I'd run this segment:

[source,sql]
----

SELECT id, data->'start_date' AS startDate, data->'average_speed' AS averageSpeed
FROM runs
WHERE jsonb_array_elements(data->'segment_efforts')->'segment'->>'id' = '6818475'

ERROR:  argument of WHERE must not return a set
LINE 3: WHERE jsonb_array_elements(data->'segment_efforts')->'segmen...
----

This doesn't work since +++<cite>+++https://www.postgresql.org/docs/9.5/static/functions-json.html#FUNCTIONS-JSON-PROCESSING-TABLE[jsonb_array_elements]+++</cite>+++ returns a set of boolean values, as http://stackoverflow.com/questions/21675174/querying-data-within-json-array-data-field/21677486#21677486[Craig Ringer points out on Stack Overflow].

Instead we can use a https://www.postgresql.org/docs/9.4/static/queries-table-expressions.html#QUERIES-LATERAL[LATERAL subquery] to achieve our goal:

[source,sql]
----

SELECT id, data->'start_date' AS startDate, data->'average_speed' AS averageSpeed
FROM runs r,
LATERAL jsonb_array_elements(r.data->'segment_efforts') segment
WHERE segment ->'segment'->>'id' = '6818475'

    id     |       startdate        | averagespeed
-----------+------------------------+--------------
 455461182 | "2015-12-24T11:20:26Z" | 2.841
 440088621 | "2015-11-27T06:10:42Z" | 2.975
 407930503 | "2015-10-07T05:18:34Z" | 2.985
 317170464 | "2015-06-03T04:44:59Z" | 2.842
 312629236 | "2015-05-27T04:46:33Z" | 2.857
 277786711 | "2015-04-02T05:25:59Z" | 2.408
 226351235 | "2014-12-05T07:59:15Z" | 2.803
 225073326 | "2014-12-01T06:15:21Z" | 2.929
 224287690 | "2014-11-29T09:02:46Z" | 3.087
 223964715 | "2014-11-28T06:18:29Z" | 2.844
(10 rows)
----

Perfect!
ReactJS/Material-UI: Cannot resolve module 'material-ui/lib/'
Date: 2017-02-12 22:43:53

I've been playing around with ReactJS and the Material-UI library over the weekend and ran into this error while trying to follow http://www.material-ui.com/v0.15.0-alpha.1/#/components/subheader[one of the example from the demo application]:

[source,text]
----

ERROR in ./src/app/modules/Foo.js
Module not found: Error: Cannot resolve module 'material-ui/lib/Subheader' in /Users/markneedham/neo/reactjs-test/src/app/modules
 @ ./src/app/modules/Foo.js 13:17-53
webpack: Failed to compile.
----

This was the component code:

[source,javascript]
----

import React from 'react'
import Subheader from 'material-ui/lib/Subheader'

export default React.createClass({
  render() {
    return <div>
    <Subheader>Some Text</Subheader>
    </div>
  }
})
----

which is then rendered like this:

[source,javascript]
----

import Foo from './modules/Foo'
render(Foo, document.getElementById("app"))
----

I came across https://github.com/callemall/material-ui/issues/4845[this post on Stack Overflow] which seemed to describe a similar issue and led me to realise that I was actually on the wrong version of the documentation. I'm using version 0.16.7 but the demo I copied from is for version 0.15.0-alpha.1!

This is the component code that we actually want:

[source,javascript]
----

import React from 'react'
import Subheader from 'material-ui/Subheader'

export default React.createClass({
  render() {
    return <div>
    <Subheader>Some Text</Subheader>
    </div>
  }
})
----

And that's all I had to change. There are several other components that you'll see the same error for and it looks like the change was made between the 0.14.x and 0.15.x series of the library.
scikit-learn: Creating a matrix of named entity counts
Date: 2017-11-29 23:01:38
Learn how to create a matrix of named entities using scikit-learn's CountVectorizer and then train a model on it as part of machine learning pipeline.

I've been trying to improve my score on Kaggle's https://www.kaggle.com/c/spooky-author-identification[Spooky Author Identification] competition, and my latest idea was building a model which used named entities extracted using the https://github.com/aboSamoor/polyglot[polyglot NLP library].

We'll start by learning how to extract entities form a sentence using polyglot which isn't too tricky:

[source,python]
----

>>> from polyglot.text import Text
>>> doc = "My name is David Beckham. Hello from London, England"
>>> Text(doc, hint_language_code="en").entities
[I-PER(['David', 'Beckham']), I-LOC(['London']), I-LOC(['England'])]
----

This sentence contains three entities. We'd like each entity to be a string rather than an array of values so let's refactor the code to do that:

[source,python]
----

>>> ["_".join(entity) for entity in Text(doc, hint_language_code="en").entities]
['David_Beckham', 'London', 'England']
----

That's it for the polyglot part of the solution. Now let's work out how to integrate that with scikit-learn.

I've been using scikit-learn's +++<cite>+++http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html[Pipeline]+++</cite>+++ abstraction for the other models I've created so I'd like to take the same approach here. This is an example of a model that creates a matrix of unigram counts and creates a Naive Bayes model on top of that:

[source,python]
----

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

nlp_pipeline = Pipeline([
    ('cv', CountVectorizer(),
    ('mnb', MultinomialNB())
])

...
# Train and Test the model
...
----

I was going to write a class similar to +++<cite>+++http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html[CountVectorizer]+++</cite>+++ but after reading its code for a couple of hours I realised that I could just pass in a custom analyzer instead. This is what I ended up with:

[source,python]
----

entities = {}


def analyze(doc):
    if doc not in entities:
        entities[doc] = ["_".join(entity) for entity in Text(doc, hint_language_code="en").entities]
    return entities[doc]

nlp_pipeline = Pipeline([
    ('cv', CountVectorizer(analyzer=lambda doc: analyze(doc))),
    ('mnb', MultinomialNB())
])
----

I'm caching the results in a dictionary because the entity extraction is quite time consuming and there's no point recalculating it each time the function is called.

Unfortunately this model didn't help me improve my best score. It scores a log loss of around 0.5, a bit worse than the 0.45 I've achieved using the unigram model :(
Python: polyglot - ModuleNotFoundError: No module named 'icu'
Date: 2017-11-28 19:52:13
Learn how to get around the ModuleNotFoundError: No module named '`icu`' error when installing Python's polyglot NLP library

I wanted to use the https://github.com/aboSamoor/polyglot[polyglot] NLP library that my colleague Will Lyon mentioned in http://www.lyonwj.com/2017/11/15/entity-extraction-russian-troll-tweets-neo4j/[his analysis of Russian Twitter Trolls] but had installation problems which I thought I'd share in case anyone else experiences the same issues.

I started by trying to install +++<cite>+++polyglot+++</cite>+++:

[source,bash]
----

$ pip install polyglot

ImportError: No module named 'icu'
----

Hmmm I'm not sure what +++<cite>+++icu+++</cite>+++ is but luckily there's a https://github.com/aboSamoor/polyglot/issues/10[GitHub issue covering this problem]. That led me to https://tobywf.com/2017/05/installing-pyicu-on-macos/[Toby Fleming's blog post] that suggests the following steps:

[source,bash]
----

brew install icu4c
export ICU_VERSION=58
export PYICU_INCLUDES=/usr/local/Cellar/icu4c/58.2/include
export PYICU_LFLAGS=-L/usr/local/Cellar/icu4c/58.2/lib
pip install pyicu
----

I already had +++<cite>+++icu4c+++</cite>+++ installed so I just had to make sure that I had the same version of that library as Toby did. I ran the following command to check that:

[source,bash]
----

$ ls -lh /usr/local/Cellar/icu4c/
total 0
drwxr-xr-x  12 markneedham  admin   408B 28 Nov 06:12 58.2
----

That still wasn't enough though! I had to install these two libraries as well:

[source,bash]
----

pip install pycld2
pip install morfessor
----

I was then able to install polyglot, but had to then run the following commands to download the files needed for entity extraction:

[source,bash]
----

polyglot download embeddings2.de
polyglot download ner2.de
polyglot download embeddings2.en
polyglot download ner2.en
----
Python 3: TypeError: unsupported format string passed to numpy.ndarray.*format*
Date: 2017-11-19 07:16:56
This post explains how to work around a change in how Python string formatting works for numpy arrays between Python 2 and Python 3.

This post explains how to work around a change in how Python string formatting works for numpy arrays between Python 2 and Python 3.

I've been going through https://twitter.com/justmarkham[Kevin Markham]'s scikit-learn Jupyter notebooks and ran into a problem on the https://github.com/justmarkham/scikit-learn-videos/blob/master/07_cross_validation.ipynb[Cross Validation] one, which was throwing this error when attempting to print the KFold example:

[source,python]
----

Iteration                   Training set observations                   Testing set observations
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-28-007cbab507e3> in <module>()
      6 print('{} {:^61} {}'.format('Iteration', 'Training set observations', 'Testing set observations'))
      7 for iteration, data in enumerate(kf, start=1):
----> 8     print('{0:^9} {1} {2:^25}'.format(iteration, data[0], data[1]))

TypeError: unsupported format string passed to numpy.ndarray.__format__
----

We can reproduce this easily:

[source,python]
----

>>> import numpy as np
----

[source,python]
----

>>> "{:9}".format(np.array([1,2,3]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported format string passed to numpy.ndarray.__format__
----

What about if we use Python 2?

[source,python]
----

>>> "{:9}".format(np.array([1,2,3]))
'[1 2 3]  '
----

Hmmm, must be a change between the Python versions.

We can work around it by coercing our numpy array to a string:

[source,python]
----

>>> "{:9}".format(str(np.array([1,2,3])))
'[1 2 3]  '
----
Neo4j Browser: Expected entity id to be an integral value
Date: 2017-11-06 16:17:35
Learn how to handle the 'Expected entity id to be an integral value' error that you get if you pass in 32 bit integer values as params to a query.

I came across an interesting error while writing a Cypher query that used parameters in the Neo4j browser which I thought I should document for future me.

We'll start with a graph that has 1,000 people:

[source,cypher]
----

unwind range(0,1000) AS id
create (:Person {id: id})
----

Now we'll try and retrieve some of those people via a parameter lookup:

[source,cypher]
----

:param ids: [0]
----

[source,cypher]
----

match (p:Person) where p.id in {ids}
return p

╒════════╕
│"p"     │
╞════════╡
│{"id":0}│
└────────┘
----

All good so far. Now what about if we try and look them up by their internal node id instead:

[source,cypher]
----

match (p:Person) where id(p) in {ids}
return p

Neo.ClientError.Statement.TypeError
Expected entity id to be an integral value
----

Hmm, that was unexpected. It turns out that to get this query to work we need to cast each of the integer values in the +++<cite>+++\{ids}+++</cite>+++ array to a 64 bit integer using the +++<cite>+++https://neo4j.com/docs/developer-manual/current/cypher/functions/scalar/#functions-tointeger[toInteger]+++</cite>+++ function:

[source,cypher]
----

match (p:Person) where id(p) in [id in {ids} | toInteger(id)]
return p

╒════════╕
│"p"     │
╞════════╡
│{"id":0}│
└────────┘
----

Success!