Add mention of e2fsprogs to runtime dependencies.

Signed-off-by: Marc Tamsky <mtamsky@gmail.com> (github: tamsky)
Add overlayfs graph backend

This backend uses the overlayfs union filesystem for containers
plus hard link file sharing for images.

Each container/image can have a "root" subdirectory which is a plain
filesystem hierarchy, or they can use overlayfs.

If they use overlayfs there is a "upper" directory and a "lower-id"
file, as well as "merged" and "work" directories. The "upper"
directory has the upper layer of the overlay, and "lower-id" contains
the id of the parent whose "root" directory shall be used as the lower
layer in the overlay. The overlay itself is mounted in the "merged"
directory, and the "work" dir is needed for overlayfs to work.

When a overlay layer is created there are two cases, either the
parent has a "root" dir, then we start out with a empty "upper"
directory overlaid on the parents root. This is typically the
case with the init layer of a container which is based on an image.
If there is no "root" in the parent, we inherit the lower-id from
the parent and start by making a copy if the parents "upper" dir.
This is typically the case for a container layer which copies
its parent -init upper layer.

Additionally we also have a custom implementation of ApplyLayer
which makes a recursive copy of the parent "root" layer using
hardlinks to share file data, and then applies the layer on top
of that. This means all chile images share file (but not directory)
data with the parent.

Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
pkg/mount: add more sharesubtree options

Signed-off-by: Vincent Batts <vbatts@redhat.com>
pkg/mount: testing for linux sharedsubtree mounts

* shared
* shared/slave
* unbindable
* private

Signed-off-by: Vincent Batts <vbatts@redhat.com>
pkg/mount: adding fields supported by freebsd

Signed-off-by: Vincent Batts <vbatts@redhat.com>
Add check for IP_NF_FILTER

Signed-off-by: Alexandr Morozov <lk4d4@docker.com>
Finalize TarSum Version 1 w/ refactor

The current Dev version of TarSum includes hashing of extended
file attributes and omits inclusion of modified time headers.

I refactored the logic around the version differences to make it
more clear that the difference between versions is in how tar
headers are selected and ordered.

TarSum Version 1 is now declared with the new Dev version continuing
to track it.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
devmapper: Move file write and rename functionality in a separate function

Currently we save device metadata and have a helper function saveMetadata()
which converts data in json format as well as saves it to file. For
converting data in json format, one needs to know what is being saved.

Break this function down in two functions. One function only has file
write capability and takes in argument about byte array of json data.
Now this function does not have to know what data is being saved. It
only knows about a stream of json data is being saved to a file.

This allows me to reuse this function to save a different type of
metadata. In this case I am planning to save NextDeviceId so that
docker can use this device Id upon next restart. Otherwise docker
starts from 0 which is suboptimal.

Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
devmapper: Export nextDeviceId so that json.Marshal() can operate on it

I was trying to save nextDeviceId to a file but it would not work and
json.Marshal() will do nothing. Then some search showed that I need to
make first letter of struct field capital, exporting this field and
now json.Marshal() works.

This is a preparatory patch for the next one.

Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
devmapper: Save and restore NextDeviceId in a file

The way thin-pool right now is designed, user space is supposed to keep
track of what device ids have already been used. If user space tries to
create a new thin/snap device and device id has already been used, thin
pool retuns -EEXIST.

Upon receiving -EEXIST, current docker implementation simply tries the
NextDeviceId++ and keeps on doing this till it finds a free device id.

This approach has two issues.

- It is little suboptimal.
- If device id already exists, current kenrel implementation spits out
  a messsage on console.

[17991.140135] device-mapper: thin: Creation of new snapshot 33 of device 3 failed.

Here kenrel is trying to tell user that device id 33 has already been used.
And this shows up for every device id docker tries till it reaches a point
where device ids are not used. So if there are thousands of container and
one is trying to create a new container after fresh docker start, expect
thousands of such warnings to flood console.

This patch saves the NextDeviceId in a file in
/var/lib/docker/devmapper/metadata/deviceset-metadata and reads it back
when docker starts. This way we don't retry lots of device ids which 
have already been used. 

There might be some device ids which are free but we will get back to them
once device numbers wrap around (24bit limit on device ids).

This patch should cut down on number of kernel warnings.

Notice that I am creating a deviceset metadata file which is a global file
for this pool. So down the line if we need to save more data we should be
able to do that.

Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Compute TarSum on storage of image layer content

Now, newly created/imported layers will have the checksum of
the layer diff computed and stored in the image json file.

For now, it is not an error if the computed checksum does not
match an existing checksum, only a warning message is logged. The
eventual goal is to use the checksums in the image JSON to verify
the integrity of the layer contents when doing `docker load` or
`docker pull`, and error out if it does not match.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
devmapper: Fix gofmt related build failures

My pull request failed the build due to gofmat issues. I have run gofmt
on specified files and this commit fixes it.

Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
devicemapper: split out devicemapper bindings

This is a first pass at splitting out devicemapper into separate, usable
bindings.

Signed-off-by: Vincent Batts <vbatts@redhat.com>
Make test-unit runnable without parallel

Signed-off-by: Yohei Ueda <yohei@jp.ibm.com>
docker-remove-redundant-json-tags

In previous patch I had introduce json:"-" tags to be on safer side to make
sure certain fields are not marshalled/unmarshalled. But struct fields
starting with small letter are not exported so they will not be marshalled
anyway. So remove json:"-" tags from there.

Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
btrfs: information for the information gods

Signed-off-by: Vincent Batts <vbatts@redhat.com>
Export envvars necessary for unit-test

Signed-off-by: Yohei Ueda <yohei@jp.ibm.com>
Cleanup api server creation

Current implementation is hard to reason about because of trying to mix
unix/tcp server implementations, even though they are quite different.
This cleans that up.

Also makes it possible to create and manage a new API server easily,
e.g. for adding an introspection socket to a container.

Built in such a way as to allow a non-HTTP server to work as well, such
as libchan.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Don't loose precision when parsing image size on 32 bit machines. Presumably fixes #8979.

Signed-off-by: Recursive Madman <recursive.madman@gmx.de>
LINKED CONTAINER ID PASSED TO LXC

This passed the --net=container:CONTINER_ID to lxc-start as --share-net
Docker-DCO-1.1-Signed-off-by: Abin Shahab <ashahab@altiscale.com> (github: ashahab-altiscale)