Doesn't it send up everything in the context directory, regardless of whether anything is ADDed?  Because otherwise, the client would have to parse the Dockerfile also, to figure out what needs to be included in said context.