Repository: bitfield/script
Pull request #40 (2020-03-17 16:15:11)

calc SHA-256 hash of file
Add a `CheckSum()` sinks func that will return the checksum of a file. (https://github.com/bitfield/script/issues/39).

The checksum is a **SHA-256** hash of the file.
The result is a `string` to be readable.

bitfield: Hey @thomaspoignant , awesome PR, thank you!

I wonder if there's also a case for treating the contents of the pipe as a list of files (as generated, for example, by `ListFiles()`), and transforming them into hash sums. That is to say, a SHA256 _filter_ as well as a _sink_, in `script` language. What do you think?

I also wonder if we should name this `SHA256`, to make it clear which algorithm is being used (there are a zillion different kinds of `CheckSum`, for example).

thomaspoignant: Thanks @bitfield for you feedback.
I have rename the sink to `SHA256`.

Just to be sure, if we use `ListFiles()` and the result is something like 
``` 
testdata/multiple_files/1.txt
testdata/multiple_files/2.txt
testdata/multiple_files/3.tar.zip
```
What do you want as the result of the sink?
Something like:
```
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
b267dc7e66ee428bc8b51b1114bd0e05bde5c8c5d20ce828fbc95b83060c2f17
```
bitfield: That looks pretty good, yes! In general the semantics seem to be that if you call a sink method (such as `String()`, you get a final result and an error, and that's the end of the pipe. That's appropriate for a single file, opened with `File()`, for example. On the other hand, if your pipe contains a list of _files_ (such as that produced by `ListFiles()`), then a filter method replaces each line with the transformed version of that line (like `DirName()`, for example).

I think it would be really useful if you could add an example program (in the `examples` directory) that shows the way someone might use this in a real-world context. For example, calculating hash sums of a bunch of files.
thomaspoignant: @bitfield sorry to ask a lot of question 🤷‍♂️

But do you think I should keep a _filter_ and a _sink_?
If I want to keep them both I can have a name conflict because I have 2 function which are doing almost the same thing.


bitfield: No need to apologise! I find discussions like this one quite useful for thrashing out how stuff should work and what makes sense to everybody.

I see your point about the naming conflict. I suppose it depends on what the example program looks like. Would we ever want to get the checksums for multiple files at once? If not, we don't need the filter, just the sink.

What do you think about `SHA256Sum()` as a name? Then if somebody wants `MD5Sum()` they could have that, and it matches the Unix command name.

By the way, one thing that occurred to me about the tests. We compare the result from `p.SHA256()` by checksumming the file ourselves using the `sha256` library... but that, of course, is how `p.SHA256()` is implemented! So we're not really testing anything here, except our ability to call a library. And if there should be a bug in that library, we won't know, will we?

A good practice with test expectations is to always calculate them yourself, by some other means, different to what you're using to implement the feature. So we could paste the contents of the test file into https://emn178.github.io/online-tools/sha256.html or some other tool, get the result as a hash, and compare that to `p.SHA256()` as a string literal.
thomaspoignant: @bitfield I made some changes, SHA256Sum is now a filter and works after a `ListFiles()`.
I added an [example](https://github.com/bitfield/script/blob/e9043cd6e32d2a742500a2912bbe7e6e045b8c01/examples/sha256sum/main.go), I hope it is clearer than before what it do.
bitfield: Looking good!

I feel like it would be good to have the ability to checksum both the contents of a pipe, as bytes, and a set of files. What do you think? If we did that then `SHA256Sum()` would work for the pipe, and maybe we could call the file version `SHA256Sums()`?

The example program is nice. How about we expand it slightly to compare the sums of two files, and tell you whether or not they're the same? I always like to make the example programs a tiny bit more interesting, if possible.
thomaspoignant: @bitfield Thanks for your time.
I have create @ method now, one `filter` and one `sink`, can you tell me if it looks alright?
thomaspoignant: @bitfield do you have any feedbacks on my updated pull request? 
Should I change something?
thomaspoignant: @bitfield I have tried to resolve all your feedbacks, and don't hesitate if you have more feedback.

Patch:
diff --git a/README.md b/README.md
index ab793ac..420ee80 100644
--- a/README.md
+++ b/README.md
@@ -113,11 +113,13 @@ John is also a [Kubernetes and cloud infrastructure consultant](https://bitfield
 	- [RejectRegexp](#rejectregexp)
 	- [Replace](#replace)
 	- [ReplaceRegexp](#replaceregexp)
+	- [SHA256Sums](#sha256Sums)
 - [Sinks](#sinks)
 	- [AppendFile](#appendfile)
 	- [Bytes](#bytes)
 	- [CountLines](#countlines)
 	- [Read](#read)
+	- [SHA256Sum](#sha256Sum)
 	- [Stdout](#stdout)
 	- [String](#string)
 	- [WriteFile](#writefile)
@@ -303,6 +305,7 @@ If you're already familiar with shell scripting and the Unix toolset, here is a
 | `find`             | [`FindFiles`](#findfiles)                                     |
 | `ls`               | [`ListFiles()`](#listfiles)                                   |
 | `sed`              | [`Replace()`](#replace) / [`ReplaceRegexp()`](#replaceregexp) |
+| `sha256sum`        | [`SHA256Sum()`](#sha256Sum) / [`SHA256Sums()`](#sha256sums)   |
 | `tail`             | [`Last()`](#last)                                             |
 | `uniq -c`          | [`Freq()`](#freq)                                             |
 | `wc -l`            | [`CountLines()`](#countlines)                                 |
@@ -735,6 +738,17 @@ p := script.File("test.txt").Replace("old", "new")
 p := script.File("test.txt").ReplaceRegexp(regexp.MustCompile("Gol[a-z]{1}ng"), "Go")
 ```
 
+## SHA256Sums
+`SHA256Sums()` reads a list of file paths from the pipe, one per line, and returns a pipe which contains the SHA-256 checksum of each file.
+If there are any errors (for example, non-existent files), the pipe's error status will be set to the first error encountered, but execution will continue.
+
+Examples:
+
+| Input                                                                                                    | `SHA256Sums` output                                                                                                                                                                                             |
+| -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `testdata/sha256Sum.input.txt`                                                                           | `1870478d23b0b4db37735d917f4f0ff9393dd3e52d8b0efa852ab85536ddad8e`                                                                                                                                             |
+| `testdata/multiple_files/1.txt`<br>`testdata/multiple_files/2.txt`<br>`testdata/multiple_files/3.tar.gz` | `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`<br>`e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`<br>`e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` |
+
 # Sinks
 
 Sinks are operations which return some data from a pipe, ending the pipeline.
@@ -766,6 +780,17 @@ var numLines int
 numLines, err := script.File("test.txt").CountLines()
 ```
 
+## SHA256Sum
+
+`SHA256Sum()`, as the name suggests, returns the [SHA256 checksum](https://en.wikipedia.org/wiki/SHA-2) of the file as a hexadecimal number stored in a string, plus an error:
+```go
+var sha256Sum string
+sha256Sum, err := script.File("test.txt").SHA256Sum()
+```
+### Why not MD5? 
+
+[MD5 is insecure](https://en.wikipedia.org/wiki/MD5#Security).
+
 ## Read
 
 `Read()` behaves just like the standard `Read()` method on any `io.Reader`:
@@ -846,6 +871,7 @@ Since `script` is designed to help you write system administration programs, a f
 * [least_freq](examples/least_freq/main.go)
 * [visitors](examples/visitors/main.go)
 * [ls](examples/ls/main.go)
+* [sha256Sum](examples/sha256sum/main.go)
 
 [More examples would be welcome!](https://github.com/bitfield/script/pulls)
 
diff --git a/examples/sha256sum/main.go b/examples/sha256sum/main.go
new file mode 100644
index 0000000..7a47eb5
--- /dev/null
+++ b/examples/sha256sum/main.go
@@ -0,0 +1,27 @@
+package main
+
+import (
+	"fmt"
+	"github.com/bitfield/script"
+	"log"
+	"os"
+	"strings"
+)
+
+func main() {
+	// Read the first 2 args, and calculate the checksum of the files.
+	hashFiles, err := script.Args().First(2).SHA256Sums().String()
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	// Compare the SHA256 checksum to check if files are identical.
+	checkSums := strings.Split(hashFiles,"\n")
+	if checkSums[0] == checkSums[1] {
+		fmt.Println("Hashes are identical")
+		os.Exit(0)
+	}
+
+	fmt.Printf("Hashes are different: %q vs %q\n", checkSums[0], checkSums[1])
+	os.Exit(1)
+}
diff --git a/filters.go b/filters.go
index 09f6544..a1ba4ce 100644
--- a/filters.go
+++ b/filters.go
@@ -4,6 +4,8 @@ import (
 	"bufio"
 	"bytes"
 	"container/ring"
+	"crypto/sha256"
+	"encoding/hex"
 	"fmt"
 	"io"
 	"os"
@@ -354,3 +356,32 @@ func (p *Pipe) ReplaceRegexp(re *regexp.Regexp, replace string) *Pipe {
 		out.WriteRune('\n')
 	})
 }
+
+// SHA256Sums reads a list of file paths from the pipe, one per line, and returns a
+// pipe which contains the SHA-256 checksum of each pathname.
+// If there are any errors (for
+// example, non-existent files), the pipe's error status will be set to the
+// first error encountered, but execution will continue.``
+func (p *Pipe) SHA256Sums() *Pipe {
+	if p == nil || p.Error() != nil {
+		return p
+	}
+
+	return p.EachLine(func(line string, out *strings.Builder) {
+		f, err := os.Open(line)
+		if err != nil {
+			p.SetError(err)
+			return
+		}
+		defer f.Close()
+
+		h := sha256.New()
+		if _, err := io.Copy(h, f); err != nil {
+			p.SetError(err)
+			return
+		}
+
+		out.WriteString(hex.EncodeToString(h.Sum(nil)[:]))
+		out.WriteRune('\n')
+	})
+}
diff --git a/sinks.go b/sinks.go
index 9c4b5f7..c32c553 100644
--- a/sinks.go
+++ b/sinks.go
@@ -1,6 +1,8 @@
 package script
 
 import (
+	"crypto/sha256"
+	"encoding/hex"
 	"fmt"
 	"io"
 	"io/ioutil"
@@ -44,6 +46,25 @@ func (p *Pipe) CountLines() (int, error) {
 	return lines, p.Error()
 }
 
+// SHA256Sum calculates the SHA-256 of the file from the pipe's reader, and returns the
+// string result, or an error. If there is an error reading the pipe, the pipe's
+// error status is also set.
+func (p *Pipe) SHA256Sum() (string, error) {
+	if p == nil || p.Error() != nil {
+		return "", p.Error()
+	}
+
+	bytes, err := p.Bytes()
+	if err != nil {
+		p.SetError(err)
+		return "", p.Error()
+	}
+
+	sum := sha256.Sum256(bytes)
+	encodedCheckSum := hex.EncodeToString(sum[:])
+	return encodedCheckSum, nil
+}
+
 // Stdout writes the contents of the pipe to the program's standard output. It
 // returns the number of bytes successfully written, plus a non-nil error if the
 // write failed or if there was an error reading from the pipe. If the pipe has


Test patch:
diff --git a/filters_test.go b/filters_test.go
index e570df3..3af8bf8 100644
--- a/filters_test.go
+++ b/filters_test.go
@@ -462,3 +462,25 @@ func TestDirname(t *testing.T) {
 		}
 	}
 }
+
+func TestSHA256Sums(t *testing.T) {
+	t.Parallel()
+	testCases := []struct {
+		testFileName string
+		want         string
+	}{
+		// To get the checksum run: openssl dgst -sha256 <file_name>
+		{"testdata/sha256Sum.input.txt", "1870478d23b0b4db37735d917f4f0ff9393dd3e52d8b0efa852ab85536ddad8e\n"},
+		{"testdata/hello.txt", "b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9\n"},
+		{"testdata/multiple_files", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\n"},
+	}
+	for _, tc := range testCases {
+		got, err := ListFiles(tc.testFileName).SHA256Sums().String()
+		if err != nil {
+			t.Error(err)
+		}
+		if got != tc.want {
+			t.Errorf("%q: want %q, got %q", tc.testFileName, tc.want, got)
+		}
+	}
+}
\ No newline at end of file
diff --git a/pipes_test.go b/pipes_test.go
index 610f48e..12a11b9 100644
--- a/pipes_test.go
+++ b/pipes_test.go
@@ -107,6 +107,10 @@ func doMethodsOnPipe(t *testing.T, p *Pipe, kind string) {
 	p.Concat()
 	action = "CountLines()"
 	p.CountLines()
+	action = "SHA256Sums()"
+	p.SHA256Sums()
+	action = "SHA256Sum()"
+	p.SHA256Sum()
 	action = "Dirname()"
 	p.Dirname()
 	action = "EachLine()"
diff --git a/sinks_test.go b/sinks_test.go
index 8a4fa64..ff17778 100644
--- a/sinks_test.go
+++ b/sinks_test.go
@@ -54,6 +54,11 @@ func doSinksOnPipe(t *testing.T, p *Pipe, kind string) {
 	if err != nil {
 		t.Error(err)
 	}
+	action = "SHA256Sum()"
+	_, err = p.SHA256Sum()
+	if err != nil {
+		t.Error(err)
+	}
 	action = "WriteFile()"
 	_, err = p.WriteFile("testdata/tmp" + kind)
 	defer os.Remove("testdata/tmp" + kind)
@@ -84,6 +89,29 @@ func TestZeroPipeSinks(t *testing.T) {
 	doSinksOnPipe(t, &Pipe{}, "zero")
 }
 
+func TestSHA256Sum(t *testing.T) {
+	t.Parallel()
+	testCases := []struct {
+		testFileName string
+		want         string
+	}{
+		{"testdata/empty.txt", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"},
+		{"testdata/test.txt", "a562c9c95e2ff3403e7ffcd8508c6b54d47d5f251387758d3e63dbaaa8296341"},
+		{"testdata/bytes.bin", "b267dc7e66ee428bc8b51b1114bd0e05bde5c8c5d20ce828fbc95b83060c2f17"},
+	}
+
+	for _, tc := range testCases {
+		p := File(tc.testFileName)
+		got, err := p.SHA256Sum()
+		if err != nil {
+			t.Error(err)
+		}
+		if got != tc.want {
+			t.Errorf("want %q, got %q", tc.want, got)
+		}
+	}
+}
+
 func TestCountLines(t *testing.T) {
 	t.Parallel()
 	want := 3
@@ -226,4 +254,4 @@ func TestBytes(t *testing.T) {
 	if !bytes.Equal(got, want) {
 		t.Errorf("want %q, got %q", want, got)
 	}
-}
+}
\ No newline at end of file
diff --git a/testdata/sha256Sum.input.txt b/testdata/sha256Sum.input.txt
new file mode 100644
index 0000000..f8e7cfe
--- /dev/null
+++ b/testdata/sha256Sum.input.txt
@@ -0,0 +1,4 @@
+apple
+orange
+banana
+banana
\ No newline at end of file