Repository: uber/makisu (go)

Issue:
With some special base images, image size drastically larger than when built using docker
New to makisu and built my first image. After successful build I find that the image is (as reported by docker images) 1.49G. That same image built with docker results in an image size of 811M.

```
FROM runatlantis/atlantis:v0.8.2

# install terragrunt binary

RUN curl -LOks https://github.com/gruntwork-io/terragrunt/releases/download/v${TERRAGRUNT_VERSION}/terragrunt_linux_amd64  && \
      mv terragrunt_linux_amd64 /usr/local/bin/terragrunt && \
      chmod +x /usr/local/bin/terragrunt && ln -s /usr/local/bin/terragrunt /usr/local/bin/terragrunt0.18

RUN curl -LOks https://github.com/gruntwork-io/terragrunt/releases/download/v${TERRAGRUNT_VERSION}/terragrunt_linux_amd64  && \
      mv terragrunt_linux_amd64 /usr/local/bin/terragrunt0.19 && \
      chmod +x /usr/local/bin/terragrunt0.19

# Versions: https://pypi.python.org/pypi/awscli#downloads
ENV AWS_CLI_VERSION 1.16.157

RUN apk update && \
    apk add python \
            py-pip py-setuptools \
            ca-certificates \
            ansible \
            jq && \
    pip --no-cache-dir install awscli==${AWS_CLI_VERSION} && \
    rm -rf /var/cache/apk/*

COPY config/aws /home/atlantis/.aws

# copy our custom plugins/providers
#COPY plugins /home/atlantis/.terraform.d/plugins

# copy updated providers
COPY bin /usr/local/bin/

# link terraform to version 0.11.14
RUN rm -f /usr/local/bin/terraform && \
    ln -s /usr/local/bin/terraform0.11.14 /usr/local/bin/terraform
```

Inspecting the resulting image and comparing that to the one that docker build produces shows different layers (both in hash and number). 

Inspect of makisu image

```
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:bcf2f368fe234217249e00ad9d762d8f1a3156d60c442ed92079fa5b120634a1",
                "sha256:32717ade1adbfd23bf4489ac1fdb862888c916eaa787322419ce87e95ed83bbf",
                "sha256:4851334180e692b67a53046bfbb1337d9e899d151f0476fbd9cdb21316fc455d",
                "sha256:f82e81ba32a7b279efcc14baf720a5a4f41e21faf5543d53ca378c1538f10b6e",
                "sha256:a723cda4dfd776bda07ccc96cca49b0bef5647586fc3c161393f3181ed3bc359",
                "sha256:0c3fb05691d830512a35ba4fec5bc96bcd104b48dc3fb649a5e6a4379b660698",
                "sha256:9b6cb2a46269f90fce16234610a28f2dee5a30f6050b0972057e29568d95f098",
                "sha256:321c00e3b0abba69b12da8e73e427e01a371c725d64eec2ffa9827dfd0c5ad85"
            ]
        }
```

Inspect of docker image

```
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:bcf2f368fe234217249e00ad9d762d8f1a3156d60c442ed92079fa5b120634a1",
                "sha256:32717ade1adbfd23bf4489ac1fdb862888c916eaa787322419ce87e95ed83bbf",
                "sha256:4851334180e692b67a53046bfbb1337d9e899d151f0476fbd9cdb21316fc455d",
                "sha256:f82e81ba32a7b279efcc14baf720a5a4f41e21faf5543d53ca378c1538f10b6e",
                "sha256:a723cda4dfd776bda07ccc96cca49b0bef5647586fc3c161393f3181ed3bc359",
                "sha256:0c3fb05691d830512a35ba4fec5bc96bcd104b48dc3fb649a5e6a4379b660698",
                "sha256:9b6cb2a46269f90fce16234610a28f2dee5a30f6050b0972057e29568d95f098",
                "sha256:4450521046838f5252bc6115485283c2796ac07000be72b38857103e209263a0",
                "sha256:9ed78fb9f246e4663814e31971db3f3968cfc37dabf2f02f99d2d6b4ff6a9d74",
                "sha256:90234a343ee2efd14fa48ac52b3aa79e0a38456d3e3ee575842c34c034a30ca9",
                "sha256:3bb3295ac4b5f5eb1adc809ebb668ab9460d57bb51a5802e1ab1120fe7263d62",
                "sha256:354c38982758323f3785a541d9f957279c4454ad67a0670ccfc412ea68ca60d5",
                "sha256:27fdcb5138781980d388d3be3363cd8964901f2e855e57abeedb3c9cb4a4ed64"
            ]
        }
```

From the looks of it the the first 7 layers are common (and are actually from the base image runatlantis/atlantis:v0.8.2). In makisu it appears that all of the layers generated from my Dockerfile are squashed into one (which would make caching difficult no?) whereas in the docker build version they are not. 

Pull request:
More granular file diff comparison logic to handle special base images
Partially fix #243 

Some base images contains file of TypeRegA instead of TypeReg. They should be treated as the same type.

Patch:
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
new file mode 100644
index 0000000..4207170
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -0,0 +1,27 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Environments**
+e.g Kuberenetes, AWS, GCP, Data centers, laptop, etc.
+
+**Additional context**
+Add any other context about the problem here--environment configurations, logs, etc.
diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md
new file mode 100644
index 0000000..bbcbbe7
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/feature_request.md
@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.
diff --git a/.travis.yml b/.travis.yml
index 5005c99..19b93f9 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -14,9 +14,19 @@ before_script:
 env:
   - GOPATH="" GO111MODULE=on
 
-script:
-  - "chmod -R a+rw . || true"
-  - "make cunit-test"
-  - "chmod -R a+rw . || true"
-  - "make integration"
-  - "fossa --option allow-unresolved:true"
+jobs:
+  include:
+    - stage: "Tests"
+      name: "Unit Tests"
+      script:
+        - "chmod -R a+rw . || true"
+        - "make cunit-test"
+    - name: "Integration Tests"
+      script:
+        - "chmod -R a+rw . || true"
+        - "make integration"
+    - stage: "Compliance"
+      name: "Fossa Check"
+      script:
+        - "chmod -R a+rw . || true"
+        - "fossa --option allow-unresolved:true"
diff --git a/README.md b/README.md
index b7a1ba8..17903fe 100644
--- a/README.md
+++ b/README.md
@@ -116,6 +116,7 @@ Makisu caches docker image layers both locally and in docker registry (if --push
 
 For example, Redis can be setup as a distributed cache key-value store with this [Kubernetes job spec](examples/k8s/redis.yaml).
 Then connect Makisu to redis cache by passing `--redis-cache-addr=redis:6379` argument.
+If the Redis server is password-protected, use `--redis-cache-password=password` argument.
 Cache has a 7 day TTL by default, which can be configured with `--local-cache-ttl=7d` argument.
 
 For more options on cache, please see [documentation](docs/CACHE.md).
diff --git a/bin/makisu/README.md b/bin/makisu/README.md
index e15a740..4bbb7df 100644
--- a/bin/makisu/README.md
+++ b/bin/makisu/README.md
@@ -20,6 +20,7 @@ Flags:
       --blacklist stringArray           Makisu will ignore all changes to these locations in the resulting docker images
       --local-cache-ttl duration        Time-To-Live for local cache (default 168h0m0s)
       --redis-cache-addr string         The address of a redis server for cacheID to layer sha mapping
+      --redis-cache-password string           The password of the Redis server, should match 'requirepass' in redis.conf
       --redis-cache-ttl duration        Time-To-Live for redis cache (default 168h0m0s)
       --http-cache-addr string          The address of the http server for cacheID to layer sha mapping
       --http-cache-header stringArray   Request header for http cache server. Format is "--http-cache-header <header>:<value>"
diff --git a/bin/makisu/cmd/build.go b/bin/makisu/cmd/build.go
index 1b25b10..47112a4 100644
--- a/bin/makisu/cmd/build.go
+++ b/bin/makisu/cmd/build.go
@@ -51,11 +51,12 @@ type buildCmd struct {
 	commit        string
 	blacklists    []string
 
-	localCacheTTL     time.Duration
-	redisCacheAddress string
-	redisCacheTTL     time.Duration
-	httpCacheAddress  string
-	httpCacheHeaders  []string
+	localCacheTTL      time.Duration
+	redisCacheAddress  string
+	redisCachePassword string
+	redisCacheTTL      time.Duration
+	httpCacheAddress   string
+	httpCacheHeaders   []string
 
 	dockerHost    string
 	dockerVersion string
@@ -109,6 +110,7 @@ func getBuildCmd() *buildCmd {
 
 	buildCmd.PersistentFlags().DurationVar(&buildCmd.localCacheTTL, "local-cache-ttl", time.Hour*168, "Time-To-Live for local cache")
 	buildCmd.PersistentFlags().StringVar(&buildCmd.redisCacheAddress, "redis-cache-addr", "", "The address of a redis server for cacheID to layer sha mapping")
+	buildCmd.PersistentFlags().StringVar(&buildCmd.redisCachePassword, "redis-cache-password", "", "The password of the Redis server, should match 'requirepass' in redis.conf")
 	buildCmd.PersistentFlags().DurationVar(&buildCmd.redisCacheTTL, "redis-cache-ttl", time.Hour*168, "Time-To-Live for redis cache")
 	buildCmd.PersistentFlags().StringVar(&buildCmd.httpCacheAddress, "http-cache-addr", "", "The address of the http server for cacheID to layer sha mapping")
 	buildCmd.PersistentFlags().StringArrayVar(&buildCmd.httpCacheHeaders, "http-cache-header", nil, "Request header for http cache server. Format is \"--http-cache-header <header>:<value>\"")
diff --git a/bin/makisu/cmd/utils.go b/bin/makisu/cmd/utils.go
index 404ceef..cdd2103 100644
--- a/bin/makisu/cmd/utils.go
+++ b/bin/makisu/cmd/utils.go
@@ -169,7 +169,7 @@ func (cmd *buildCmd) newCacheManager(buildContext *context.BuildContext, imageNa
 	if cmd.redisCacheAddress != "" {
 		log.Infof("Using redis at %s for cacheID storage", cmd.redisCacheAddress)
 
-		kvStore, err = keyvalue.NewRedisStore(cmd.redisCacheAddress, cmd.redisCacheTTL)
+		kvStore, err = keyvalue.NewRedisStore(cmd.redisCacheAddress, cmd.redisCachePassword, cmd.redisCacheTTL)
 		if err != nil {
 			log.Errorf("Failed to connect to redis store: %s", err)
 		}
diff --git a/docs/CACHE.md b/docs/CACHE.md
index c47bf57..374e7bd 100644
--- a/docs/CACHE.md
+++ b/docs/CACHE.md
@@ -21,6 +21,7 @@ To disable it, set ttl to 0s.
 To configure redis cache, use the following options:
 ```
 --redis-cache-addr string         The address of a redis server for cacheID to layer sha mapping
+--redis-cache-password string           The password of the Redis server, should match 'requirepass' in redis.conf
 --redis-cache-ttl duration        Time-To-Live for redis cache (default 168h0m0s)
 ```
 
diff --git a/lib/cache/keyvalue/redis_store.go b/lib/cache/keyvalue/redis_store.go
index 9d498b8..c76c62d 100644
--- a/lib/cache/keyvalue/redis_store.go
+++ b/lib/cache/keyvalue/redis_store.go
@@ -36,13 +36,14 @@ type redisStore struct {
 // NewRedisStore returns a new instance of Store backed by a redis server.
 // In this constructor we try to open a connection to redis. If that attempt fails
 // we return an error. If it succeeds we just close that connection.
-func NewRedisStore(addr string, ttl time.Duration) (Store, error) {
+func NewRedisStore(addr, password string, ttl time.Duration) (Store, error) {
 	cli := redis.NewClient(&redis.Options{
 		Addr:         addr,
 		MaxRetries:   MaxRetires,
 		DialTimeout:  DialTimeout,
 		ReadTimeout:  ReadTimeout,
 		WriteTimeout: WriteTimeout,
+		Password:     password,
 	})
 	if _, err := cli.Ping().Result(); err != nil {
 		return nil, err
diff --git a/lib/tario/compare.go b/lib/tario/compare.go
index ea1352c..50e3959 100644
--- a/lib/tario/compare.go
+++ b/lib/tario/compare.go
@@ -22,10 +22,6 @@ import (
 
 // IsSimilarHeader returns if the given headers are describing similar entries.
 func IsSimilarHeader(h *tar.Header, nh *tar.Header) (bool, error) {
-	if h.Typeflag != nh.Typeflag {
-		return false, nil
-	}
-
 	// Don't support modifying "/".
 	if h.Name == "" && nh.Name == "" {
 		return true, nil
@@ -33,12 +29,24 @@ func IsSimilarHeader(h *tar.Header, nh *tar.Header) (bool, error) {
 
 	switch h.Typeflag {
 	case tar.TypeSymlink:
+		if nh.Typeflag != tar.TypeSymlink {
+			return false, nil
+		}
 		return isSimilarSymlink(h, nh)
 	case tar.TypeLink:
+		if nh.Typeflag != tar.TypeLink {
+			return false, nil
+		}
 		return isSimilarHardLink(h, nh)
 	case tar.TypeDir:
+		if nh.Typeflag != tar.TypeDir {
+			return false, nil
+		}
 		return isSimilarDirectory(h, nh)
 	case tar.TypeReg, tar.TypeRegA:
+		if nh.Typeflag != tar.TypeReg && nh.Typeflag != tar.TypeRegA {
+			return false, nil
+		}
 		return isSimilarRegularFile(h, nh)
 	default:
 		return false, fmt.Errorf("unsupported type %b", h.Typeflag)


Test patch:
diff --git a/lib/cache/keyvalue/redis_store_test.go b/lib/cache/keyvalue/redis_store_test.go
index dc7e9a7..fcb3ab5 100644
--- a/lib/cache/keyvalue/redis_store_test.go
+++ b/lib/cache/keyvalue/redis_store_test.go
@@ -32,7 +32,7 @@ func TestRedisStore(t *testing.T) {
 
 		d, err := time.ParseDuration("10s")
 		require.NoError(err)
-		store, err := NewRedisStore(s.Addr(), d)
+		store, err := NewRedisStore(s.Addr(), "", d)
 		require.NoError(err)
 
 		loc, err := store.Get("a")
@@ -48,7 +48,7 @@ func TestRedisStore(t *testing.T) {
 
 		d, err := time.ParseDuration("10s")
 		require.NoError(err)
-		store, err := NewRedisStore(s.Addr(), d)
+		store, err := NewRedisStore(s.Addr(), "", d)
 		require.NoError(err)
 
 		defer store.Cleanup()
diff --git a/lib/tario/compare_test.go b/lib/tario/compare_test.go
index 9eb14d6..068ee86 100644
--- a/lib/tario/compare_test.go
+++ b/lib/tario/compare_test.go
@@ -507,6 +507,42 @@ func TestIsSimilarRegularFile(t *testing.T) {
 		require.NoError(err)
 	})
 
+	t.Run("NoChangeWithTypeRegA", func(t *testing.T) {
+		require := require.New(t)
+
+		tmpRoot, err := ioutil.TempDir("/tmp", "makisu-test")
+		require.NoError(err)
+		defer os.RemoveAll(tmpRoot)
+
+		testFile1, err := ioutil.TempFile(tmpRoot, "test1")
+		require.NoError(err)
+		testFile2, err := ioutil.TempFile(tmpRoot, "test2")
+		require.NoError(err)
+
+		atime := time.Now().Add(-time.Hour)
+		mtime := time.Now().Add(-time.Hour * 2)
+		require.NoError(os.Chtimes(testFile1.Name(), atime, mtime))
+		require.NoError(os.Chtimes(testFile2.Name(), atime, mtime))
+
+		fi1, err := os.Lstat(testFile1.Name())
+		require.NoError(err)
+		fi2, err := os.Lstat(testFile2.Name())
+		require.NoError(err)
+
+		h, err := tar.FileInfoHeader(fi1, "")
+		require.NoError(err)
+		require.Equal(tar.TypeReg, int32(h.Typeflag))
+		newH, err := tar.FileInfoHeader(fi2, "")
+		require.NoError(err)
+		newH.Typeflag = tar.TypeRegA
+		similar, err := isSimilarRegularFile(h, newH)
+		require.True(similar)
+		require.NoError(err)
+		similar, err = IsSimilarHeader(h, newH)
+		require.True(similar)
+		require.NoError(err)
+	})
+
 	t.Run("DifferentContentButSameSizeConsideredSimilar", func(t *testing.T) {
 		require := require.New(t)