markhneedham/blog huggingface dataset:
Ruby: Ignore header line when parsing CSV file
Date: 2008-10-04 01:32:08

As my Ruby journey continues one of the things I wanted to do today was parse a CSV file.

http://www.rubytips.org/2008/01/06/csv-processing-in-ruby/[This article] proved to be very useful for teaching the basics but it didn't say how to ignore the header line that the CSV file contained.

The CSV file I was parsing was similar to this:

[source,text]
----

name, surname, location
Mark, Needham, Sydney
David, Smith, London
----

I wanted to get the names of people originally to use them in my code. This was the first attempt:

[source,ruby]
----

require 'csv'

def parse_csv_file_for_names(path_to_csv)
  names = []
  csv_contents = CSV.read(path_to_csv)
  csv_contents.each do |row|
    names << row[0]
  end
  return names
end
----

I then printed out the names to see what was going on:

[source,ruby]
----

names = parse_csv_file_for_names( "csv_file.csv" )
names.each do |name|
  puts name
end
----

This is what was printed:

[source,text]
----

name
Mark
David
----

It turns out that the 'shift' method is what I was looking for to help me ignore the first line of the file. The new and improved method now looks like this:

[source,ruby]
----

require 'csv'

def parse_csv_file_for_names(path_to_csv)
  names = []
  csv_contents = CSV.read(path_to_csv)
  csv_contents.shift
  csv_contents.each do |row|
    names << row[0]
  end
  return names
end
----

Not a particularly complicated thing to do in the end although I had been expecting to find a method on CSV that would allow me to ignore the header line automatically. As far as I could tell there isn't one!
Ruby LDAP Options
Date: 2008-10-05 16:29:32

As I http://www.markhneedham.com/blog/2008/09/29/connecting-to-ldap-server-using-opends-in-java/[mentioned in an earlier post] a colleague and I spent a few days looking at how to connect to an http://www.opends.org/[OpenDS] LDAP server using Ruby.

We ended up analysing four different solutions for solving the problem.

== Active LDAP

This approach involved using the http://rubyforge.org/projects/ruby-activeldap/[Active LDAP Ruby] which "provides an object oriented interface to LDAP. It maps LDAP entries to Ruby objects with LDAP attribute accessors based on your LDAP server's schema and each object's objectClasses".

We had real problems trying to even connect to our OpenDS server using this library. We eventually found out that OpenDS is not actually listed as one of the supported interfaces.

The real benefit of this approach was that the library is written in Ruby meaning that getting permission to install it would be easier.

The fact that we couldn't actually get it to work didn't help!

== Java LDAP libraries + RJB

This approach involved interacting with http://www.markhneedham.com/blog/2008/09/29/connecting-to-ldap-server-using-opends-in-java/[LDAP with Java libraries] and then using the http://rjb.rubyforge.org/[Ruby Java Bridge] to connect to these from our Ruby code.

We were able to solve the problem quite easily using this approach but the Ruby code we ended up writing was very Javaesque in style and it didn't feel like we were utilising the power of Ruby by using Java for such a fundamental part of the problem we were attempting to solve.

On the positive side RJB is easily installable via a gem and we were able to connect to OpenDS and execute the operations that were required.

== Ruby-LDAP

The third option we looked at was http://ruby-ldap.sourceforge.net/[Ruby-LDAP], a Ruby extension library written in C.

The disadvantage of this was that we needed to have make available to install it onto our machine. Seeing as we were using a Mac this meant downloading XCode to make use of the GCC compiler.

Interacting with the different libraries was tricky initially but we eventually got the hang of it and were able to connect to OpenDS despite it not being listed as one of the supported libraries.

== ruby-net-ldap

http://rubyfurnace.com/docs/ruby-net-ldap-0.0.4/[ruby-net-ldap] is a pure Ruby LDAP library, installable via a gem.

This had by far the best examples and most intuitive interface of the options that we analysed and worked for us first time without too much fuss. Connecting to our Open DS server was seamless.

== Overall

Our original selection, despite the slightly more complicated installation was Ruby-LDAP.

However, http://olabini.com/[Ola Bini] pointed out ruby-net-ldap which actually proved to meet our criteria even more closely than Ruby-LDAP did and as such was the option we went with.

For those that are interested, http://geekdamana.blogspot.com/[Damana] has posted more of the http://geekdamana.blogspot.com/2008/10/ruby-ldap.html[technical details] behind the approaches we took.
Ignore file in Svn
Date: 2008-10-02 21:10:27

I spent a bit of time this afternoon marveling at the non intuitiveness of working out how to ignore files in Svn.

Normally I'd just use http://tortoisesvn.tigris.org/[Tortoise SVN] as it makes it so easy for you but I really wanted to know how to do it from the shell!

After a bit of Googling and conversation with a http://pilchardfriendly.wordpress.com/[colleague] I think I have it figured out to some extent.

== Ignoring just one file or pattern

If you only have one pattern or file that you want to ignore then the following command should do the trick.

[source,text]
----

svn propset svn:ignore <file_or_pattern_to_ignore> <dir_in_which_to_create_ignore_file>
----

For example:

[source,text]
----

svn propset svn:ignore build .
----

This means my 'build' directory will now be ignored and the svn ignore file will be placed in the current directory.

== Ignoring multiple files or patterns

The problem with the above approach comes when you want to ignore more than one pattern/file. If you just run the propset command again it overrides the current svn ignore file with the current value - clearly not what we want!

Luckily propedit comes to the rescue.

Running the following command will open up your chosen editor and allow you to edit the svn ignore file.

[source,text]
----

svn propedit svn:ignore <dir_where_ignore_file_resides>
----

When I initially did this I received the following error:

[source,text]
----

svn: None of the environment variables SVN_EDITOR, VISUAL or EDITOR is set, and no 'editor-cmd' run-time configuration option was found
----

I wanted my default editor to be http://macromates.com/[Textmate] so I entered the following:

[source,text]
----

export SVN_EDITOR=mate
----

This didn't seem to work for me - the svn tmp file being opened up in Textmate was always empty for some reason. Changing my editor to vi seemed to fix the problem.

[source,text]
----

export SVN_EDITOR=vi
----

Running the command now opens up vi and allowed me to add the pattern '*.log' to my ignore list. If it is added successfully the following message will show up on exiting vi:

[source,text]
----

Set new value for property 'svn:ignore' on '.'
----

== Seeing which files or patterns are currently ignored

While having my Textmate problems detailed above my colleague pointed out the propget command which shows you which files/patterns are currently being ignored.

[source,text]
----

svn propget svn:ignore .
----

Running this command shows me the following:

[source,text]
----

build
*.log
----

http://svnbook.red-bean.com/en/1.4/svn-book.html#svn.advanced.props.special.ignore[svnbook] has even more goodness on ignoring files for those that are interested.
Ruby: Unzipping a file using rubyzip
Date: 2008-10-02 00:04:22

In the world of Ruby I've been working on a script which needs to unzip a file and then run an installer which is only available after unpacking it.

We've been using the http://rubyzip.sourceforge.net/[rubyzip] gem to do so but so far it hasn't felt intuitive to me coming from the Java/C# world.

http://rubyzip.sourceforge.net/classes/Zip/ZipFile.html[ZipFile] is the class we need to use and at first glance I had thought that it would be possible to just pass the zip file name to the 'extract' method and have it do all the work for me!

Turns out you actually need to open the zip file and then create the directory location for each file in the zip before extracting them all individually.

We eventually ended up with this little method:

[source,ruby]
----

require 'rubygems'
require 'zip/zip'

def unzip_file (file, destination)
  Zip::ZipFile.open(file) { |zip_file|
   zip_file.each { |f|
     f_path=File.join(destination, f.name)
     FileUtils.mkdir_p(File.dirname(f_path))
     zip_file.extract(f, f_path) unless File.exist?(f_path)
   }
  }
end
----

Which we can then call with the zip file and the destination where we want to unzip the file.

[source,ruby]
----

unzip_file("my.zip", "marks_zip")
----

Is there a better way to do this? It feels a bit clunky to me at the moment.
Testing Hibernate mappings: Where to test from?
Date: 2008-10-27 22:55:15

I've had the opportunity to work with http://www.hibernate.org/[Hibernate] and it's .NET twin http://sourceforge.net/projects/nhibernate[NHibernate] on several of my projects and one of the more interesting decisions around its use is working out the best way to test the hibernate mappings that hook together our domain model and the database.

There are three decisions to make around how best to do this:

. Where to test the mappings from?
. How to test for equality?
. How to setup the test data?

This post will focus on the ways I have seen with regards to choosing *where to test the mappings from*.

== Functional Tests

This approach advocates only testing whether we have setup the mappings correctly when we run our acceptance or functional tests - we do not write tests specifically for testing hibernate mappings.

The benefit of this approach is that we are more likely to have acceptance tests in place, so this is just another thing that they can be used to catch.

While this approach is better than not testing at all, from my experiences the test feedback cycle is too slow - it takes too long to change one of the hibernate mappings and then run the test to check if it worked or not.

== Repository Tests

With this approach we test whether our hibernate mappings are working as part of our http://domaindrivendesign.org/discussion/messageboardarchive/Repositories.html[repository] tests.

The tricky thing with testing our hibernate mappings this way is that typically we only want to set up one object in the database and then test that Hibernate hydrates it correctly, but our repository doesn't necessarily need a method for finding a single object.

We either end up adding on a method just for testing or we have to try and find our object from a list of other objects and then test it.

On the other hand, this approach does seem to work quite well when we have quite chatty repositories which provide a degree of flexibility around how we can retrieve our objects.

== Direct Tests

This approach is my current favourite and involves loading the object directly from the http://www.hibernate.org/hib_docs/v3/api/org/hibernate/Session.html[Hibernate session] and then testing it.

I was introduced to this idea by a http://www.flickr.com/photos/adsphoto/[colleague] of mine and it seems to fit the idea of testing just one thing more closely than the other two approaches.

The strange thing about this approach is that we are testing directly with an API that is hidden from our system beyond the Repository.

In terms of simplicity with regards to testing hibernate mappings, however, this is the best approach I have seen.

'''

I did a quick survey of some people last week and the most popular way of testing the mappings expressed was using Repository tests.

This post covers the other ways I have seen - are there any others people have come across or are using?
buildr - using another project's dependencies
Date: 2008-10-26 20:54:18

Through my continued use of http://incubator.apache.org/buildr[buildr] on my current project one thing we wanted to do last week was to run our production code tests using some code from the test-utilities project along with its dependencies.

I thought this would be the default behaviour but it wasn't. Looking at the http://incubator.apache.org/buildr/building.html[documentation] suggested we could achieve this by calling 'compile.dependencies' on the project, but from what I can tell you still need to explicitly state that you want to use the main test utilities code as well.

The following code in our buildfile does the job:

[source,text]
----

DEPENDENCY_JAR='depedency:dependency:jar:1.0' # change this to whatever the path to the dependency is
...
define "test.utilities" do
  compile.with DEPENDENCY_JAR
  package(:jar)
end

define "main.code" do
  # Some other code
  test.with project("test.utilities"), project("test.utilities").compile.dependencies
  package(:jar)
end
----

It seems a bit verbose but it achieves our objective in a cleaner way than having to repeat test-utilities dependencies when we run main.code's tests.

I'm sure there must be an even cleaner way than this but I'm not yet aware of it!
If you use an 'if' you deserve to suffer
Date: 2008-10-21 07:19:56

One of the things I dislike the most when coding is writing if statements. and while I don't believe that if should be completely abolished from our toolkit, I think the http://www.cuberick.com/2007/11/down-with-if.html[anti if campaign] started about a year ago is going along the right lines.

While there is certainly value in using an if statement as a http://c2.com/cgi/wiki?GuardClause[guard block] it usually feels that we have missed an abstraction if we are using it elsewhere.

Given my dislike of the construct, when I have to use it I like to being very explicit about the scope in which it is applicable. I think this approach stems from being caught out back in my university days and trying to debug something along the lines of:

[source,java]
----

if(thisHappens())
	doThis();
	doThat();
----

The code was much messier than that but I spent ages trying to work out why the correct behaviour wasn't happening. Eventually I realised that 'doThat()' was being called every time regardless of the value returned by 'thisHappens()'.

Ever since then I have written if statements like so:

[source,java]
----

if(thisHappens()) {
	doThis();
	doThat();
}
----

It doesn't look as nice but with one glance at the code I can see exactly what is happening. I definitely prefer to have this advantage although I do appreciate that if there is only one statement following the if statement then http://c2.com/xp/YouArentGonnaNeedIt.html[YAGNI] might be applicable.

I prefer to take the more conservative approach - once bitten, twice shy.

So what is the title of this post all about?

Often when pair programming there is some discussion over which approach is better and in one such session last week a colleague of mine, who happened to favour the more verbose approach, came up with a phrase along those lines when I asked why they favoured that approach. If you're going to use an if statement then you're just going to have to put up with the eye sore those extra curly braces create in your code!

Being in favour of this approach already I really like the idea and while maybe not quite as scientific as describing the technical reasons for doing so, it is perhaps more effective.
Testing Hibernate mappings: Setting up test data
Date: 2008-10-30 23:24:14

Continuing with my mini Hibernate mappings series, this post talks about the different ways of *setting up the test data* for our Hibernate tests.

. http://www.markhneedham.com/blog/2008/10/27/testing-hibernate-mappings-where-to-test-from/[Where to test the mappings from?]
. http://www.markhneedham.com/blog/2008/10/29/testing-hibernate-mappings-testing-equality/[How to test for equality?]
. How to setup the test data?

There are a couple of ways that we can setup data for Hibernate tests.

== Insert Hibernate Object

This approach involves creating a new object and saving it to the database using the save method on the http://www.hibernate.org/hib_docs/v3/api/org/hibernate/Session.html[Hibernate session].

The benefit of this approach is that our test is pretty clean. It looks like any old unit test where we create an object and then check that what we get back is the same object.

The problem is that we are effectively testing two things - the ability to save and then load our Hibernate object. We will be able to tell whether or not our Hibernate mappings are correct using this approach, but the failures we get when they do fail may not be that obvious - it could just be a database exception which makes the test fail.

== SQL Insertion

The other approach I have seen is to write manual JDBC calls to insert data into the various tables in our database and then check that we can load our object from the database using Hibernate.

The advantage of this is that our test is now only testing the mappings when we load the data from the database which helps reduce our test's invariants or potential points of failure.

The disadvantage is that the tests can seem very brittle - if we make a small change to the column names in our tables then the test setup code may now fail to work anymore.

I'm not really completely happy with either of these approaches - neither seems optimal to me but both can help us achieve our objective. It's just a matter of choosing which trade off we've prepared to accept.
Calling shell script from ruby script
Date: 2008-10-06 20:12:49

Damana and I http://geekdamana.blogspot.com/2008/10/ruby-ldap.html[previously] http://www.markhneedham.com/blog/2008/10/05/ruby-ldap-options/[posted] about our experiences with different Ruby LDAP solutions.

Having settled on http://sourceforge.net/projects/ruby-ldap/[Ruby-LDAP] (although having read Ola and Steven's comments we will now look at http://rubyforge.org/projects/net-ldap/[ruby-net-ldap]) we then needed to put together the setup, installation and teardown into a ruby script file.

A quick bit of Googling revealed that we could use the http://ruby-doc.org/core/classes/Kernel.html#M005979[Kernel.exec] method to do this.

For example, you could put the following in a ruby script file and it would execute and show you the current directory listing:

[source,ruby]
----

exec "ls"
----

The problem with using Kernel.exec, which we became aware of after reading http://blog.jayfields.com/2006/06/ruby-kernel-system-exec-and-x.html[Jay's post], is that we lose control of the current process - i.e. the script will exit after running 'exec' and won't process any other commands that follow it in the file.

Luckily for us there is another method called http://ruby-doc.org/core/classes/Kernel.html#M005982[Kernel.system] which allows us to execute a command in a sub shell, and therefore continue processing other commands that follow it.

We were able to use this method for making calls to the make script to install Ruby-LDAP:

[source,ruby]
----

@extconf = "ruby extconf.rb"
system @extconf
system "make"
system "make install"
----

There is one more option we can use if we need to collect the results  called %x[...]. We didn't need to collect the results so we have gone with 'Kernel.system' for the time being.

Jay http://blog.jayfields.com/2006/06/ruby-kernel-system-exec-and-x.html[covers the options in more detail on his post] for those that need more information than I have presented.
rspec - Invalid character '\240' in expression
Date: 2008-10-06 20:48:48

We have been using http://rspec.info/[rspec] on my project for the unit testing of our Ruby code and while running one of the specs last week I ended up getting this somewhat en-cryptic error message:

[source,text]
----

Invalid character '\240' in expression
...
----

After convincing myself that this error wasn't actually possible it turned out that I had somehow entered an 'invisible to TextMate' character after one of the method definitions - on the editor it just looked like a space.

I'm not sure exactly what the character was but deleting it got rid of the error and got my test/spec running again.
Browsing around the Unix shell more easily
Date: 2008-10-15 22:31:16

Following on from my post about getting the http://www.markhneedham.com/blog/2008/09/28/show-pwd-all-the-time/[pwd to display on the bash prompt all the time] I have learnt a couple of other tricks to make the shell experience more productive.

http://en.wikipedia.org/wiki/Unalias#Removing_aliases_.28unalias.29[Aliases] are the first new concept I came across and several members of my current team and I now have these setup.

We are primarily using them to provide a shortcut command to get to various locations in the file system. For example I have the following 'work' alias in my ~/.bash_profile file:

[source,text]
----

alias work='cd ~/path/to/my/current/project'
----

I can then go to the bash prompt and type 'work' and it navigates straight there. You can put as many different aliases as you want in there, just don't forget to execute the following command after adding new ones to get them reflected in the current shell:

[source,text]
----

. ~/.bash_profile
----

A very simple idea but one that helps save so many keystrokes for me every day.

Another couple of cool commands I recently discovered are *pushd* and *popd*

They help provide a stack to store directories on, which I have found particularly useful when browsing between distant directories.

For example suppose I am in the directory '/Users/mneedham/Desktop/Blog/' but I want to go to '/Users/mneedham/Projects/Ruby/path/to/some/code' to take a look at some code.

Before changing to that directory I can execute:

[source,text]
----

pushd .
----

This will push the current directory ('/Users/mneedham/Desktop/Blog/') onto the stack. Then once I'm done I just need to run:

[source,text]
----

popd
----

I'm back to '/Users/mneedham/Desktop/Blog/' with a lot less typing.

Running the following command shows a list of the directories currently on the stack:

[source,text]
----

dirs
----

I love navigating with the shell so if you've get any other useful tips please share them!
Tomcat - No caching of RESTlet resources for Firefox
Date: 2008-10-22 22:00:33

One problem that we've been trying to solve today is how to make a http://www.restlet.org/[RESTlet] resource non cacheable.

The reason for this is that when a user logs out of the system and then hits the back button they shouldn't be able to see that page, but instead should see the login form.

After several hours of trawling Google and trying out various different suggestions we http://www.experts-exchange.com/Software/Server_Software/Web_Servers/Apache/Q_20880931.html?qid=20880931[came across the idea] of setting 'cache-control' with the value 'no-store' in the response headers.

The code to make this happen is as follows (use inside a class which extends http://www.restlet.org/documentation/1.0/api/org/restlet/resource/Resource.html[Resource]):

[source,java]
----

HttpResponse response = (HttpResponse) getResponse();
Series<Parameter> headers = response.getHttpCall().getResponseHeaders();
headers.add("cache-control", "no-store");
----

The important part in this example is the last line. As long as it's added to the Http Response Headers that response should no longer be cached.

A bit of research revealed that Internet Explorer may change the 'no-store' value to 'no-cache' so I'm not sure if this will work for that browser.