markhneedham/blog huggingface dataset:
MySQL: Repairing broken tables/indices
Date: 2013-04-06 17:26:20

I part time administrate http://www.soccer24-7.com/forum/index.php[a football forum] that I used to run when I was at university and one problem we had recently was that some of the tables/indices had got corrupted when MySQL crashed due to a lack of disc space.

We weren't seeing any visible sign of a problem in any of the logs but whenever you tried to query one of the topics it wasn't returning any posts.

I eventually came across a useful article which explained http://www.softwareprojects.com/resources/programming/t-how-to-fix-mysql-database-myisam-innodb-1634.html[how to check whether some of the tables in a MySQL database had been corrupted and how to fix them].

I first shutdown the database using the following command:

[source,text]
----

mysqladmin shutdown
----

And then I ran this command to check on the status of each of the tables:

[source,text]
----

for path in `ls /var/lib/mysql/forum/*.MYI`; do echo $path; myisamchk $path; done
----

This gave an output like the following for each table:

[source,text]
----

Checking MyISAM file: /var/lib/mysql/forum/forum.MYI
Data records:     217   Deleted blocks:       4
myisamchk: warning: 1 client is using or hasn't closed the table properly
- check file-size
- check record delete-chain
- check key delete-chain
- check index reference
- check data record references index: 1
- check record links
MyISAM-table '/var/lib/mysql/forum/forum.MYI' is usable but should be fixed
----

If you pass the '--recover' flag to +++<cite>+++myisamchk+++</cite>+++ it will attempt to fix any problems it finds. I therefore ran the following command:

[source,text]
----

for path in `ls /var/lib/mysql/forum/*.MYI`; do echo $path; myisamchk --recover $path; done
----

After I'd run that it seemed to fix most of the problems we'd been experiencing. There are still a couple of edge cases left but at least the majority of the forum is now in a usable state.

I think we could just as easily run +++<cite>+++myisamchk+++</cite>+++ by passing a wildcard selection of files for it to run against but I didn't realise that until afterwards!

The following would therefore work just as well:~~~ ~~~text myisamchk --recover /var/lib/mysql/forum/*.MYI ~~~
Capistrano: Deploying to a Vagrant VM
Date: 2013-04-13 11:17:37

I've been working on a tutorial around thinking through problems in graphs using my football graph and I wanted to deploy it on a local http://www.vagrantup.com/[vagrant] VM as a stepping stone to deploying it in a live environment.

My Vagrant file for the VM looks like this:

[source,text]
----

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant::Config.run do |config|
  config.vm.box = "precise64"

  config.vm.define :neo01 do |neo|
    neo.vm.network :hostonly, "192.168.33.101"
    neo.vm.host_name = 'neo01.local'
    neo.vm.forward_port 7474, 57474
    neo.vm.forward_port 80, 50080
  end

  config.vm.box_url = "http://files.vagrantup.com/precise64.box"

  config.vm.provision :puppet do |puppet|
    puppet.manifests_path = "puppet/manifests"
    puppet.manifest_file  = "site.pp"
    puppet.module_path = "puppet/modules"
  end
end
----

I'm port forwarding ports 80 and 7474 to 50080 and 57474 respectively so that I can access the web app and neo4j console from my browser.

There is a bunch of puppet code to configure the machine in the location specified.

Since the web app is written in Ruby/Sinatra the easiest deployment tool to use is probably capistrano and I found the http://guides.beanstalkapp.com/deployments/deploy-with-capistrano.html[tutorial on the beanstalk website] really helpful for getting me setup.

My +++<cite>+++config/deploy.rb+++</cite>+++ file which I've got Capistrano setup to read looks like this:

[source,text]
----

require 'capistrano/ext/multistage'

set :application, "thinkingingraphs"
set :scm, :git
set :repository,  "git@bitbucket.org:markhneedham/thinkingingraphs.git"
set :scm_passphrase, ""

set :ssh_options, {:forward_agent => true}
set :default_run_options, {:pty => true}
set :stages, ["vagrant"]
set :default_stage, "vagrant"
----

In my +++<cite>+++config/deploy/vagrant.rb+++</cite>+++ file I have the following:

[source,text]
----

set :user, "vagrant"
server "192.168.33.101", :app, :web, :db, :primary => true
set :deploy_to, "/var/www/thinkingingraphs"
----

So that IP there is the same one that I assigned in Vagrantfile. If you didn't do that then you'd need to use 'vagrant ssh' to go onto the VM and then 'ifconfig' to grab the IP instead.

I figured there was probably another step required to tell Capistrano where it should get the vagrant public key from but I thought I'd try and deploy anyway just to see what would happen.

[source,text]
----

$ bundle exec cap deploy
----

It asked me to enter the vagrant user's password which is 'vagrant' by default and I eventually found http://stackoverflow.com/questions/10353530/is-there-a-way-to-deploy-into-a-vagrant-vm-using-capistrano[a post on StackOverflow] which suggested changing the 'ssh_options' to the following:

[source,text]
----

set :ssh_options, {:forward_agent => true, keys: ['~/.vagrant.d/insecure_private_key']}
----

And with that the deployment worked flawlessly! Happy days.
Sublime: Overriding default file type/Assigning specific files to a file type
Date: 2013-05-05 00:03:17

I've been using link:[Sublime] a bit recently and one thing I wanted to do was put link:[neo4j cypher] queries into files with arbitrary extensions and have them recognised as cypher files every time I open them.

I'm using the link:[cypher Sublime plugin] to get the syntax highlighting but since I've got my cypher in a .haml file it only remembers that it should have cypher highlighting as long as the file is open.

As soon as I close and then re-open the file it goes back to being highlighted as HAML.

I initially thought that the way around this would be to write a plugin which kept track of files that you'd manually assigned a syntax to but then I came across the https://github.com/facelessuser/ApplySyntax[ApplySyntax] plugin which seems even better.

ApplySyntax allows you to assign syntaxes to files based on regular expression matching on the file name or on the first line of the file.

At the moment, the easiest way to detect that a file is a cypher query is that the first line will begin with 'START' so I wrote the following in my user settings file:

_~/Library/Application Support/Sublime Text 2/Packages/User/ApplySyntax.sublime-settings_

[source,json]
----

{
	"reraise_exceptions": false,
	"new_file_syntax": false,
	"syntaxes": [
		{			
			"name": "Cypher",
			"rules": [
				{"first_line": "^START"}
			]
		}	
	]
}
----

ApplySyntax is a pretty neat plugin, worth having a look if you have this problem to solve!
A/B Testing: User Experience vs Conversion
Date: 2013-05-18 20:18:50

I've written http://www.markhneedham.com/blog/2013/01/27/ab-testing-thoughts-so-far/[a] http://www.markhneedham.com/blog/2013/04/28/ab-testing-reporting/[couple] of posts over the last few months about my experiences with A/B testing and one conversation we often used to have was around user experience vs conversion rate.

Once you start running an A/B test it encourages you to focus more on the conversion rate of users in different parts of the flow and *your inclination is to make changes that increase that conversion rate*.

Another one of our drivers is to provide the best user experience that we can to our customers and since sometimes this means that the best thing for them is not to switch it seems that these two must be in conflict.

I found it particularly interesting seeing how the conversion rate could be *impacted by the way that information was displayed to a user*.

This was an idea that I first came across when reading about http://kylerush.net/blog/optimization-at-the-obama-campaign-ab-testing/[how the Obama campaign used A/B testing] where they noticed big changes in conversion rates by making small tweaks to sentences and imagery.

Our goal from a user experience perspective was to put all the information in front of the user so that they could make an informed choice about what to do.

Initially we made the negative features of the plans very prominent and had them in a large font which led to a drop in conversion.

We assumed that people were now giving more importance to the negative features than was warranted e.g. some plans had a cancellation fee but it typically only accounted for 5% of the saving they'd make by switching to the plan.

When the product is a bit more complicated we could argue that we improve the user experience by helping the user to make an appropriate choice.

On a website the way that we do this is by how we display information by changing the font size, font weight, positioning and a variety of other things.

It's an interesting balance to find between the two drivers but if we veer towards conversion at all costs then although we'll get a higher conversion rate in the long term we'll have some frustrated customers who won't use our website again.

If we look at it that way then the two drivers don't seem so opposed to each other.
Pomodoros: Just start the timer
Date: 2013-05-27 13:23:36

I wrote earlier in the year about http://www.markhneedham.com/blog/2012/01/22/playing-around-with-pomodoros/[my use of pomodoros to track what I'm doing outside of work] and having done this for 6 months I noticed that I'm now procrastinating over picking something off the list to work on.

_(I know...I am awesome!)_

I'm not sure whether this is because I don't have anything really appealing on the list of things to work on or whether having so many things listed (I have 8-10 items) is causing the paralysis.

Either way it's a bit frustrating to achieve nothing in an evening because of something so ridiculous.

To outwit myself I decided to press the 'Start' button on the timer one day without picking something to work on and hoped that this would snap me out of my paralysis.

It's a relatively simple tweak but somehow seeing the time count down has the required effect and I've usually picked what I'm going to do within 30 seconds or so.

So if you are as good at procrastinating as me maybe you can try this too!
Polyglot Persistence: Embrace the ETL
Date: 2013-05-27 00:11:23

Over the past few years I've seen the emergence of http://martinfowler.com/bliki/PolyglotPersistence.html[polyglot persistence] i.e. using different data storage technologies for different data and in most situations we work that out up front.

image::{{<siteurl>}}/uploads/2013/05/etl1.png[Etl1,233]

For example we might use http://www.mongodb.org/[MongoDB] to store data about a customer journey through our website but we might simultaneously write page view data through to something like http://hadoop.apache.org/[Hadoop] or http://aws.amazon.com/redshift/[Redshift]:

This works reasonably well but sometimes it might not be immediately obvious how we want to query our data when we first start collecting it and our storage choice might not be the best for writing these queries.

An interesting thing to think about at this stage is whether it makes sense to add a stage to our data processing pipeline where we write an http://en.wikipedia.org/wiki/Extract,_transform,_load[ETL] job to get it into a more appropriate format:

image::{{<siteurl>}}/uploads/2013/05/etl2.png[Etl2,454]

My initial experience doing this was when I created the http://skillsmatter.com/podcast/home/what-do-you-want-to-know[ThoughtWorks graph] which involved transforming data into a graph so that I could find links between people.

https://twitter.com/A5HOK[Ashok] and I followed a similar approach for a client we went on to work for and it allowed us to find the answers to questions that couldn't be answered when the data was in its original format.

The main down side to this approach is that we now have to keep two data sources in sync but it's interesting to think about whether this trade off is worthwhile if it helps us gain new insights or find the answers to questions more quickly.

I don't have any experience with how this approach plays out over time so I'd be interesting in hearing how people have got on with this approach/if it does or doesn't work.
Viewing the contents of an archive
Date: 2013-05-29 11:22:35

Everyone now and then I want to check the contents of an archive without unpacking it and I tend to use +++<cite>+++http://linux.about.com/od/commands/l/blcmdl1_unzip.htm[unzip]+++</cite>+++ to do so:

[source,bash]
----

$ unzip -l batch-import-jar-with-dependencies.jar | tail -n 10
     1645  02-17-13 01:03   org/neo4j/batchimport/StdOutReport.class
     3089  02-17-13 01:03   org/neo4j/batchimport/structs/NodeStruct.class
     1244  02-17-13 01:03   org/neo4j/batchimport/structs/Property.class
     1732  02-17-13 01:03   org/neo4j/batchimport/structs/PropertyHolder.class
     1635  02-17-13 01:03   org/neo4j/batchimport/structs/Relationship.class
      905  02-17-13 01:03   org/neo4j/batchimport/utils/Chunker.class
     1884  02-17-13 01:03   org/neo4j/batchimport/utils/Params.class
     4445  02-17-13 01:03   org/neo4j/batchimport/Utils.class
 --------                   -------
 49947859                   16447 files
----

It does the job although it does print out some information that we're not really interested in so I was intrigued to see that https://twitter.com/apcj[Alistair] used +++<cite>+++http://linux.about.com/library/cmd/blcmdl1_zipinfo.htm[zipinfo]+++</cite>+++ when he wanted to achieve a similar thing:

[source,bash]
----

$ zipinfo -1 batch-import-jar-with-dependencies.jar | tail -n 10
org/neo4j/batchimport/ParallelImporter.class
org/neo4j/batchimport/Report.class
org/neo4j/batchimport/StdOutReport.class
org/neo4j/batchimport/structs/NodeStruct.class
org/neo4j/batchimport/structs/Property.class
org/neo4j/batchimport/structs/PropertyHolder.class
org/neo4j/batchimport/structs/Relationship.class
org/neo4j/batchimport/utils/Chunker.class
org/neo4j/batchimport/utils/Params.class
org/neo4j/batchimport/Utils.class
----

From a bit of man page reading it sounds like zipinfo is unzip, but with different flags that give an output that's a cross between unzip and ls:

____
The format is a cross between Unix ``ls -l'' and ``unzip -v'' output. See DETAILED DESCRIPTION below. Note that zipinfo is the same program as unzip (under Unix, a link to it); on some systems, however, zipinfo support may have been omitted when unzip was compiled.
____

As long as I remember I'll be using zipinfo from now on!
Ruby/Python: Constructing a taxonomy from an array using zip
Date: 2013-05-19 22:44:40

As I http://www.markhneedham.com/blog/2013/05/19/neo4jcypher-keep-longest-path-when-finding-taxonomy/[mentioned in my previous blog post] I've been hacking on a product taxonomy and I wanted to create a 'CHILD' relationship between a collection of categories.

For example, I had the following array and I wanted to transform it into an array of 'SubCategory, Category' pairs:

[source,python]
----

taxonomy = ["Cat", "SubCat", "SubSubCat"]
# I wanted this to become [("Cat", "SubCat"), ("SubCat", "SubSubCat")
----

In order to do this we need to zip the first 2 items with the last which I found reasonably easy to do using Python:

[source,python]
----

>>> zip(taxonomy[:-1], taxonomy[1:])
[('Cat', 'SubCat'), ('SubCat', 'SubSubCat')]
----

Here we using the http://stackoverflow.com/questions/509211/the-python-slice-notation[python array slicing notation] to get all but the last item of 'taxonomy' and then all but the first item of 'taxonomy' and zip them together.

I wanted to achieve that effect in Ruby though because my import job was written in that!

We can't achieve the open ended slicing as far as I can tell so the following gives us an error:

[source,ruby]
----

> taxonomy[..-1]
SyntaxError: (irb):10: syntax error, unexpected tDOT2, expecting ']'
taxonomy[..-1]
           ^
	from /Users/markhneedham/.rbenv/versions/1.9.3-p327/bin/irb:12:in `<main>'
----

The way negative indexing works is a bit different so to remove the last item of the array we use '-2' rather than '-1':

[source,ruby]
----

> taxonomy[0..-2].zip(taxonomy[1..-1])
=> [["Cat", "SubCat"], ["SubCat", "SubSubCat"]]
----
Unix: Working with parts of large files
Date: 2013-05-19 21:44:03

https://twitter.com/digitalstain[Chris] and I were looking at the neo4j log files of a client earlier in the week and wanted to do some processing of the file so we could ask the client to send us some further information.

The log file was over 10,000 lines long but the bit of the file we were interesting in was only a few hundred lines.

I usually use Vim and the ':set number' when I want to refer to line numbers in a file but Chris showed me that we can achieve the same thing with e.g. 'less -N data/log/neo4j.0.0.log'.

We can then operate on say lines 10-100 by passing the '-n' flag to sed:

____
-n By default, each line of input is echoed to the standard output after all of the commands have been applied to it. The -n option suppresses this behavior.
____

[source,bash]
----

$ sed -n '10,15p' data/log/neo4j.0.0.log
INFO: Enabling HTTPS on port [7473]
May 19, 2013 11:11:52 AM org.neo4j.server.logging.Logger log
INFO: No SSL certificate found, generating a self-signed certificate..
May 19, 2013 11:11:53 AM org.neo4j.server.logging.Logger log
INFO: Mounted discovery module at [/]
May 19, 2013 11:11:53 AM org.neo4j.server.logging.Logger log
----

We then used a combination of grep, awk and sort to work out which log files we needed.
Polyglot Persistence: The 'boring' relational option
Date: 2013-05-26 23:29:12

I was chatting with https://twitter.com/gurrie09[Brian Blignaut] last week after the http://nitetech230513.eventbrite.co.uk/?mkt_tok=3RkMMJWWfF9wsRoksq3BZKXonjHpfsX76uUtUaKg38431UFwdcjKPmjr1YAAT8R0aPyQAgobGp5I5FEPTbPYW69ut6ULXg%3D%3D[Equal Experts NoSQL event] and he made an interesting observation that in this age of http://martinfowler.com/bliki/PolyglotPersistence.html[Polyglot Persistence] we often rule out the relational database.

I think it's definitely better that we now have many different options for where we store our data - be it as key/value pairs, documents or as a network/graph.

Having these options forces us to think more about how we're going to read/write data in our application whereas previously our effort was focused around which tables we were going to pull out.

Having said that, I realised I'd fallen into the trap that Brian was referring to when thinking through how we could model the energy plans that users were selecting from a results table.

We wanted to run aggregate queries over the data to work out the most popular plans and suppliers on different days and then across different customer segments.

We represented each user selection as an event, stored as a document in http://www.mongodb.org/[MongoDB] which worked fine to start with but queries started to become much slower as the number of documents approached 500,000 or so.

I started thinking about whether we'd be better off storing the data in a different store which was better optimised for the types of queries that we wanted to run.

My initial thought was to use one of the flashier 'NoSQL' databases but https://twitter.com/a5HOK[Ashok] pointed out that the use case - slicing/dicing data at a scale (< 1 m rows) where everything would fit on disc - was perfect for something like http://www.postgresql.org/[PostgreSQL].

I realised that he was absolutely right and I think it's important to remember that in future - when we talk about *choosing the appropriate data store for our problem that doesn't mean we have to rule out the relational database* even though it's probably more fun to use another store.
neo4j/cypher: 400 response - Paths can't be created inside of foreach
Date: 2013-05-31 00:37:57

In the neo4j 1.9 milestone releases if we wanted to create multiple relationships from a node we could use the following cypher syntax:

[source,ruby]
----

require 'neography'

neo = Neography::Rest.new

neo.execute_query("create (me {name: 'Mark'})")

query =  " START n=node:node_auto_index(name={name})"
query << " FOREACH (friend in {friends} : CREATE f=friend, n-[:FRIEND]->f)"

neo.execute_query(query, {"name" => "Mark",
                          "friends" => [{ "name" => "Will"}, {"name" => "Paul"}]})
----

To check that the 'FRIEND' relationships have been created we'd write the following query:

[source,cypher]
----

START p = node:node_auto_index(name="Mark")
MATCH p-[:FRIEND]-f
RETURN f
----

[source,text]
----

==> +----------------------+
==> | f                    |
==> +----------------------+
==> | Node[2]{name:"Will"} |
==> | Node[3]{name:"Paul"} |
==> +----------------------+
==> 2 rows
==> 37 ms
----

In between the M04 and RC2 releases there was a change in the cypher parser which meant that this approach doesn't work anymore:

[source,text]
----

/Users/markhneedham/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/neography-1.0.10/lib/neography/connection.rb:168:in `handle_4xx_500_response':
Paths can't be created inside of foreach (Neography::SyntaxException)
	from /Users/markhneedham/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/neography-1.0.10/lib/neography/connection.rb:143:in `return_result'
	from /Users/markhneedham/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/neography-1.0.10/lib/neography/connection.rb:126:in `evaluate_response'
	from /Users/markhneedham/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/neography-1.0.10/lib/neography/connection.rb:45:in `post'
	from /Users/markhneedham/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/neography-1.0.10/lib/neography/rest/cypher.rb:19:in `query'
	from /Users/markhneedham/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/neography-1.0.10/lib/neography/rest.rb:335:in `execute_query'
	from create.rb:12:in `<main>'
----

Instead we can just use +++<cite>+++http://docs.neo4j.org/chunked/stable/query-create.html#create-create-multiple-nodes-from-map[CREATE]+++</cite>+++ to achieve the same thing:

[source,ruby]
----

require 'neography'

neo = Neography::Rest.new

neo.execute_query("create (me {name: 'Mark'})")

query =  " START n = node:node_auto_index(name={name})"
query << " CREATE friend = {friends} "
query << " CREATE n-[:FRIEND]->friend"

neo.execute_query(query, {"name" => "Mark",
                          "friends" => [{ "name" => "Will"}, { "name" => "Paul" }]})
----