markhneedham/blog huggingface dataset:
F#: Downloading a file from behind a proxy
Date: 2009-07-11 03:20:25

I've been continuing working on a little script to http://www.markhneedham.com/blog/2009/07/08/f-parsing-cruise-build-data/[parse Cruise build data] and the latest task was to work out how to download my Google Graph API created image onto the local disk.

I'm using the http://msdn.microsoft.com/en-us/library/system.net.webclient(VS.80).aspx[WebClient] class to do this and the  code looks like this:

[source,ocaml]
----

let DownloadGraph (fileLocation:string) (uri:System.Uri) = async {
    let webClient = new WebClient()
    webClient.DownloadFileAsync(uri, fileLocation)}
----

Sadly this doesn't work when I run it from the client site where I have access to the build metrics as there is a corporate proxy sitting in the way.

I tried Googling how to do this but all the ways that I tried kept resulting in the following error:

[source,text]
----

407 proxy authentication required
----

Even though I was entering a user name and password!

I didn't succeed until my colleague showed me a way of getting past the proxy in C# which I could quite easily use in my code:

[source,ocaml]
----

let DownloadGraph (fileLocation:string) (uri:System.Uri) = async {
    let webClient = new WebClient()
    webClient.Proxy <- new WebProxy("proxyName:port", true, null, new NetworkCredential("userName", "password", "corporateDomain"))
    webClient.DownloadFileAsync(uri, fileLocation)}
----

One thing I was doing wrong was putting a 'http' at the start of the proxyName which I think in my case was wrong as I later learn that the proxy isn't a HTTP one.

I'm also making use of http://www.infoq.com/articles/pickering-fsharp-async[asynchronous workflows] in this example so that the actual downloading of the files will be done away from the main thread - this also gives me the option to download multiple files asynchronously if I want to.
F#: Passing command line arguments to a script
Date: 2009-07-16 07:40:18

I've been doing a bit of refactoring of http://www.markhneedham.com/blog/2009/07/12/f-a-day-writing-a-feedburner-graph-creator/[my FeedBurner application] so that I can call it from the command line with the appropriate arguments and one of the problems I came across is working out how to pass arguments from the command line into an http://www.markhneedham.com/blog/2009/06/09/f-useful-for-scripting/[F# script].

With a compiled application we are able to make use of the 'http://www.markhneedham.com/blog/2009/05/02/f-entry-point-of-an-application/[EntryPointAttribute]' to get access to the arguments passed in:

[source,ocaml]
----

[<EntryPointAttribute>]
let main args =
    ShowFeedBurnerStats args
    0
----

Sadly this doesn't work with a script but http://cs.hubfs.net/forums/thread/2911.aspx[it was pointed out on Hub FS] that we can get access to all the command line arguments by using 'Sys.argv' or 'System.Environment.GetCommandLineArgs()' which seems to be the preferred choice of the compiler.

The problem is that with that method you get every single argument passed to the command line and there are some that we don't care about given the way you would typically call an F# script:

[source,text]
----

fsi --exec --nologo CreateFeedBurnerGraph.fsx -- "scotthanselman" "2009-03-01" "2009-07-14"
----

Results in the following arguments:

[source,text]
----

fsi
--exec
--nologo
CreateFeedBurnerGraph.fsx
--
scotthanselman
2009-03-01
2009-07-14
----

We care about everything after the '--' so I wrote a little function to just gather those values:

[source,ocaml]
----

    let GetArgs initialArgs  =
        let rec find args matches =
            match args with
            | hd::_ when hd = "--" -> List.to_array (matches)
            | hd::tl -> find tl (hd::matches)
            | [] -> Array.empty
        find (List.rev (Array.to_list initialArgs) ) []
----

I'm not sure this works for every possible case (if you put '--' in as an argument it wouldn't work as expected!) but it's doing the job so far.

An even better way of doing this which I came across while writing this is to use 'fsi.CommandLineArgs' which allows you to just get the arguments passed to the script. Even with this approach though the '--' is still counted as one of the arguments so the function above still makes sense.

[source,ocaml]
----

GetArgs [|"--"; "scotthanselman"; "2009-03-01"; "2009-07-14"|]
----

[source,ocaml]
----

val it : string array = [|"scotthanselman"; "2009-03-01"; "2009-07-14"|]
----

And from the script I have the following:

[source,ocaml]
----

let programArgs = fsi.CommandLineArgs |> GetArgs
ShowFeedBurnerStats programArgs
----
F#: Wrapping .NET library calls
Date: 2009-07-12 12:11:46

I've been spending a bit of time writing some code to parse the xml of my Feedburner RSS feed and create a graph to show both the daily and weekly average subscribers which you can't currently get from the Feedburner dashboard.

One thing which I found while doing this is that calls to the .NET base class library don't seem to fit in that well with the way that you would typically compose functions together in F#.

For example one of the first things I wanted to do was print the date and the circulation count to the console which I originally did like this:

[source,ocaml]
----

open System.IO
open System.Net
open Microsoft.FSharp.Control
open System.Xml.Linq
open System

let xName value = XName.Get value

// GetXml is a function of type string -> string

let GetFeedBurnerStats url =
    let feedBurnerXml = GetXml url |> XDocument.Parse
    feedBurnerXml.Descendants(xName "entry") |>
    Seq.map (fun x -> x.Attribute(xName "circulation"), x.Attribute(xName "date")) |>
    Seq.iter (fun x -> printfn "%s %s" (fst x).Value (snd x).Value)
----

It's quite annoying that we need to store the XDocument as a value before being able to call one of the methods on it to get the data that we want.

I realised that if I created a function which took in the element whose descendants I wanted to find and the XDocument I could then call the 'XDocument.Descendants()' method inside that function:

[source,ocaml]
----

let xName value = XName.Get value
let GetDescendants element (xDocument:XDocument)  = xDocument.Descendants(xName element)

let GetFeedBurnerStats =
    GetXml >>
    XDocument.Parse >>
    GetDescendants "entry" >>
    Seq.map (fun x -> x.Attribute(xName "circulation"), x.Attribute(xName "date")) >>
    Seq.iter (fun x -> printfn "%s %s" (fst x).Value (snd x).Value)
----

Since we no longer need to store the intermediate step of creating the XDocument we can now just chain together the functions using the http://www.markhneedham.com/blog/2009/01/12/f-partial-function-application-with-the-function-composition-operator/[functional composition operator] instead of the http://www.markhneedham.com/blog/2009/01/06/f-forward-operator/[forward operator].

We can also do this with the calls to 'Attribute' in the 'Seq.map' function on line 9 which helps simplify the code around there.

[source,ocaml]
----

let xName value = XName.Get value
let GetDescendants element (xDocument:XDocument)  = xDocument.Descendants(xName element)
let GetAttribute element (xElement:XElement) = xElement.Attribute(xName element)

let GetFeedBurnerStats =
    GetXml >>
    XDocument.Parse >>
    GetDescendants "entry" >>
    Seq.map (fun x -> GetAttribute "circulation" x, GetAttribute "date" x) >>
    Seq.iter (fun x -> printfn "%s %s" (fst x).Value (snd x).Value)
----
F#: Values, functions and DateTime
Date: 2009-07-25 14:10:45

One of the things I've noticed recently in my playing around with F# is that when we decide to http://www.markhneedham.com/blog/2009/07/12/f-wrapping-net-library-calls/[wrap calls] to the .NET DateTime methods there is a need to be quite careful that we are wrapping those calls with an F# function and not an F# value.

If we don't do this then the DateTime method will only be evaluated once and then return the same value for every call which is probably not the behaviour we're looking for.

The following shows how we could wrap call to get the current time in string format inside a value:

[source,ocaml]
----

let timeNow = System.DateTime.Now.ToLongTimeString()
----

If we then execute 'timeNow' to show the current time before and after a sleep this is what we see:

[source,ocaml]
----

printfn "The time now is %s" timeNow
System.Threading.Thread.Sleep(2000)
printfn "The time now is %s" timeNow
----

[source,text]
----

The time now is 2:00:29 PM
The time now is 2:00:29 PM
----

As we can see the time has remained the same despite the fact that we put a sleep in between thew two calls.

Looking at the C# version of this code via http://www.red-gate.com/products/reflector/[Reflector] we can see that 'timeNow' is just a string:

[source,csharp]
----

public string timeNow;
----

The way to get the real current time is to define a function to get the time so that it will be reevaluated each time we ask for the time:

[source,ocaml]
----

let timeNowUpToDate () = System.DateTime.Now.ToLongTimeString()
----

If we do the same test as we did above:

[source,ocaml]
----

printfn "The time now is %s" (timeNowUpToDate())
System.Threading.Thread.Sleep(2000)
printfn "The time now is %s" (timeNowUpToDate())
----

We get the following results:

[source,ocaml]
----

The time now is 2:05:13 PM
The time now is 2:05:15 PM
----

Which is what we were looking for in the first place!

Via Reflector again this is what that code would look like in C#:

[source,csharp]
----

[Serializable]
internal class timeNowUpToDate@127 : FastFunc<Unit, string>
{
    // Methods
    internal timeNowUpToDate@127()
    {
    }

    public override string Invoke(Unit unitVar0)
    {
        return DateTime.Now.ToLongTimeString();
    }
}
----

As we can see every time the function is invoked a call to the DateTime API will be made which is what we want to happen.
TDD: Testing with generic abstract classes
Date: 2009-09-18 00:40:09

In a post I wrote earlier in the week I described http://www.markhneedham.com/blog/2009/09/13/tdd-testing-sub-classes/[a dilemma we were having testing some code which made use of abstract classes] and Perryn Fowler, Liz Keogh and Pat Maddox pointed out that a useful approach for this problem would be to make use of an http://c2.com/cgi-bin/wiki?AbstractTestCases[abstract test class].

The idea here is that we create an equivalent hierarchy to our production code for our tests which in the example that I provided would mean that we have roughly the following setup:

[source,csharp]
----

public abstract class ParentModelTest
{
}

public class BusinessProcess1ModelTest : ParentModelTest { }
public class BusinessProcess2ModelTest : ParentModelTest { }
public class BusinessProcess3ModelTest : ParentModelTest { }
----

We then want to create an abstract method on the 'ParentModelTest' class to create the object under test and we'll implement this method in the sub classes.

We can then put the common tests into the abstract class and they will be run when we run the tests for each of the sub class tests.

My colleague Matt Dunn and I were looking at how you would implement this and he pointed out that it provided quite a nice opportunity to use generics to achieve our goal.

If we make the abstract test class take in generic parameters then it allows us to create our specific type in the 'create' method rather than having to create 'ParentModel' and then casting that for our specific sub class tests.

[source,csharp]
----

public abstract class ParentModelTest<T> where T : ParentModel
{
	protected abstract T CreateModel();	

	[Test]
	public void ShouldTestThoseCommonDependencies()
	{
		var model = CreateModel();
		// Do some awesome testing that's common across all 3 sub classes here
	}
}
----

[source,csharp]
----

[TestFixture]
public class BusinessProcessModel1Test : ParentModelTest<BusinessProcessModel1>
{
	protected override BusinessProcessModel1 CreateModel()
	{
		return new BusinessProcessModel1(...);
	}

	[Test]
	public void ShouldDoSomeEquallyOutstandingStuff()
	{
		var model = CreateModel();
		// and test away
	}
}
----

It seems to work quite well and we realised that it's actually a pattern that we had also used to test that our controllers would actually be injected with all their dependencies from our DI container or whether we have forgotten to register some of them.

We have an abstract class similar to this which all our controller tests extend:

[source,csharp]
----

public abstract class ContainerTest<T> where T : Controller
{
	[Test]
	public void ShouldHookUpAllDependencies()
	{
		var container = CreateTheContainer();

		var controller = container.Resolve<T>();

		Assert.IsNotNull(controller);
	}
}
----

[source,csharp]
----

public class AwesomeControllerTest : ContainerTest<AwesomeController> { }
----

When we haven't got all the dependencies injected correctly the code will actually blow up on the resolve step so the last line is not strictly necessary but the test seems like it's not testing anything at first glance without it.
Impersonators: Using them in showcases
Date: 2009-09-10 00:23:33

Towards the end of http://blog.rufiao.com/2009/08/impersonator-pattern/[my colleague Julio Maia's blog post about the impersonator pattern] he suggests that the standalone environment that we can create through the use of impersonators can be quite useful for showcases and we actually had a recent occasion where we had to switch mid-showcase from using the integration environment to make use of an impersonator.

In this case part of the environment went down in the middle of the showcase so *if we wanted to keep on going then that was our only option* but in general the  expectation of the business is that our showcases show them the functionality of the application end to end.

We need to be careful when using a standalone environment that the business are aware of this because it's quite likely that the application is going to respond much more quickly than it would if it was fully integrated.

If people aren't aware of this then they will be very surprised when the application isn't quite as performant when we put it into a real environment and it has to retrieve and store real data instead of just getting pre-stored values from an in memory impersonator.

Another disadvantage of using a standalone environment is that we must ensure that we only enter data which we have previously entered otherwise we'll either go against the real environment if we're using a http://martinfowler.com/bliki/SelfInitializingFake.html[self initializing fake] or (I think) just get no response if we're using a record/replay type impersonator.

If we're letting a business user drive the application during a showcase then we'll need to ensure that they know which data they can enter.

As long as we're careful in those regards I think it can be quite a useful approach since *we can make our showcase independent of factors that our outside of our control* - for example in our most recent showcase we had to reschedule the time because one part of the server was going to be restarted at our initial showcase time.

I definitely like the idea of using the impersonator for the safety it brings although my thinking at the moment is that maybe it's best to use it as a backup option if the real environments aren't working.

I'd be interested to hear if others have experiences around this area.
A reminder that sometimes it's best just to ask
Date: 2009-09-07 22:27:57

Recently my pair and I were trying to merge some changes into our code that we had just picked up fron updating from the trunk and realised that we weren't actually sure how to resolve that merge since it seemed to conflict with what we'd been working on.

We hadn't checked in for longer than we would have liked to due to a bit of a checkin pile up which had happened because the build on the CI server had been failing for a few hours due to a temporary problem we were having with  an external dependency.

Another pair were working on some code which was around the same area of the code base as us but for a different feature so we hadn't been working together too closely.

Usually when faced with this type of problem my default approach would be to go to the Subversion logs and go through each of the changes in their checkin until I could work out what their change was all about and how it fitted in with what we were doing.

On this occasion we didn't do that but instead called over one of the guys in the pair to ask if they could explain what changes that made in their latest checkin and how we would need to change the code on our machine to work with those changes.

My colleague took us to a whiteboard and sketched out what they were working on and the approach they were taking to achieve that.

It took less than a minute to do this and after that it was really obvious what changes we need to make and in another couple of minutes we'd made those changes and had our code functioning again.

It's certainly a useful skill to be able to merge code by scanning through the change logs but it's also useful to remember that software development is generally a team game and that we can get things done so much quicker if we work closely with our team mates.
Fiddler: Trying to work out how it all hooks together
Date: 2009-09-06 23:25:42

I http://www.markhneedham.com/blog/2009/06/24/using-fiddler-with-iis/[mentioned previously] that we're making use of Fiddler quite a lot on my current project, mainly to check the traffic going to and from the service layer, and I'm quite curious how it actually works.

In particular I wanted to know:

* How we're able to route requests through Fiddler and then through the corporate proxy
* How proxy settings work differently for Firefox and Internet Explorer

As far as I'm aware the source code for Fiddler isn't available so a colleague and I tracked the various proxy settings when Fiddler was turned on and off and also had a look at some registry settings.

image::{{<siteurl>}}/uploads/2009/09/fiddler.png[fiddler.png,673]

As Internet Explorer is a http://msdn.microsoft.com/en-us/library/aa383630%28VS.85%29.aspx[WinInet] based application it looks at the Local Area Network settings to check whether there are any proxies that it needs to route requests through. I believe any proxy defined in here would be known as the 'system proxy'.

We noticed that our application also routed its requests via the proxy defined in the LAN settings as well - it is a .NET application but I guess that even applications written in other languages but running on windows would make use of the system proxy too?

Firefox has its own proxy settings which are accessible via the preferences menu of the browser and are then configurable from the 'Advanced \-> Network' tab.

If another proxy is defined in the LAN settings when Fiddler starts up it takes this value and stores it somewhere (in memory presumably) as the 'Upstream gateway proxy' (Tools \-> Fiddler Options \-> Connections) which it directs requests through before they go out to the internet.

It then changes the proxy defined in the LAN settings to point to itself - typically port 8888 locally.

The 'Upstream gateway proxy' always seems to be the same as the 'system proxy' so if a different proxy is defined on FireFox than Internet Explorer this won't be used even though on the latest version of Fiddler when it starts up the FireFox proxy will be changed to point to Fiddler.

When Fiddler is shut down it reverts the proxy settings to how they were previously although (unsurprisingly) if it crashes then we are left in a state where requests are being sent to 'http://localhost:8888' and there is no application listening on that port anymore meaning that we end up with a 'Cannot find server or DNS Error'.

Restarting Fiddler gets rid of that problem although the 'Upstream gateway proxy' usually seems to get lost so if we want to make requests out to the internet then we need to go and reset that in our LAN settings before restarting Fiddler.

If there's anything inaccurate in what I've written or if you know of any resources that would help me understand this better if you let me know in the comments that'd be cool!
TDD: Test only constructors
Date: 2009-09-12 00:35:12

I http://www.markhneedham.com/blog/2009/09/06/coding-checking-invariants-in-a-factory-method/[wrote previously how we'd been doing some work to change the way that we get a 'User' object into our system] and one mistake that we made intially was to have another constructor on the 'User' object which was being used in all our unit tests which involved the user in some way.

The original reason that this 'test constructor' was created was to make it easier to construct a 'fake user' which we were using in some of our functional tests but had ended up being used in unit tests as well.

The constructor being exposed for testing was pretty much the same as the private constructor that we have now.

~~~csharp

public class User
{
	...
	public User(string userName, string customerId)
	{
		this.userName = userName;
		this.customerId = customerId;
	}
}~~~
The problem with this approach to testing is that we aren't actually testing the code which we're using in production so even if the tests pass it doesn't actually tell us very much about our code.

We could be doing everything right in this test constructor and doing something crazy in the static factory method and we wouldn't find out until much later on.

The interesting thing is that the method that we call from the production code isn't as testing friendly as the one we had made public just for testing:

~~~csharp

....
public static User CreateUserFrom(NameValueCollection headers)
{
	var userName = headers["user-name-header-tag"];
	var customerId = headers["customer-id-header-tag"];

	// and so on

	return new User(userName, customerId);
}~~~ In order to make our test setup code use this method we had to create a NameValueCollection containing key/value pairs with the appropriate keys that reside in the headers of requests coming into our application.
....

We therefore end up with code similar to this in the test data builder:

~~~csharp

public class UserBuilder
{
	...

 public User Build()
 {
 	var headers = new NameValueCollection();
 	headers.Add("user-name-header-tag", "randomName");
 	headers.Add("customer-id-header-tag", "customerId");
 	User.CreateUserFor(headers);
 } }~~~ This leaks a bit of the implementation of 'CreateUserFrom' into the tests but I prefer this to testing something which is never actually used.