Repository: pyrossh/rust-embed (language: rust)

Issue:
Feature Request: Include some file metadata w/ the file
I'm using rust-embed to create an executable that serves some files over HTTP.

HTTP has support for metadata, like the [Last-Modified] date, and [ETag], which can help browser caches.

[Last-Modified]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Last-Modified
[ETag]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag

If rust-embed included the last-modified date, as well as some cryptographic hash (say, sha256 and/or sha512?), then I could use that data to serve Last-Modified and ETag headers to help browsers more effectively cache these resources.

In the meantime, I suppose I could work around the issue by calculating content hashes in memory. It'd be handy to have them at compile time, though.


Discussion / hints:
@pyros2097 what do you think of this? We could probably have a separate function on `RustEmbed` which exposes the metadata, like:
```rust
fn metadata(path: &str) -> Option<FileMetadata>;
```

`FileMetadata` would then be an opaque struct so we could add more fields to it without breaking backwards compatibility.

It should probably go behind a feature flag (then we wouldn't even need a new major release, just minor).
Sounds great!

> It should probably go behind a feature flag (then we wouldn't even need a new major release, just minor).

I think additive changes are fine in minor releases even without a feature flag. (<https://semver.org/>) But, a feature flag might be nice so folks don't incur any additional processing time that they don't opt into. 
Oh right, yeah either way we wouldn't need a major release.

Edit: though, we would need a major release if there is no default implementation for the trait function. Otherwise it would be an error if the user didn't implement the function.
@AzureMarker this looks great. Let's do a major release. I think it makes sense to expose this to everyone since most users use it for websites (including me) and the extra meta would really help.
It might be nice if that `FileMetadata` type also returns the same `Cow<'static, [u8]>` that you get from `get(file_path)`, that way you don't have to make a call to both `get(path)` and `metadata(path)`.

```rust
if let Some(metadata) = my_embed.metadata(path) {
   do_something(&metadata.bytes, &metadata.sha512);
}

// vs:
match (my_embed.get(path), my_embed.metadata(path)) {
   (Some(bytes), Some(metadata)) => {
      do_something(&bytes, &metadata.sha512);
   }, 
   (None, None) => { /* do nothing */ },
   _ => { panic!("This should never happen?"); }
}
```
Since we are releasing a new major version, does it make sense to change the api of `get` to return a File object,
```rust
fn get(path: &str) -> Option<File>;
```

That way we have all the fields like name, size, meta under it so that it could be useful for other cases.

It would also be a bonus if we could update our examples to add http caching in them.
```golang
package main

import (
  "net/http"

  "github.com/pyros2097/go-embed/examples/assets"
)

func main() {
  http.HandleFunc("/", func(res http.ResponseWriter, req *http.Request) {
    println("GET " + req.URL.Path)
    data, hash, contentType, err := assets.Asset("public/", req.URL.Path)
    if err != nil {
      data, hash, contentType, err = assets.Asset("public", "/index.html")
      if err != nil {
        data = []byte(err.Error())
      }
    }
    res.Header().Set("Content-Encoding", "gzip")
    res.Header().Set("Content-Type", contentType)
    res.Header().Add("Cache-Control", "public, max-age=31536000")
    res.Header().Add("ETag", hash)
    if req.Header.Get("If-None-Match") == hash {
      res.WriteHeader(http.StatusNotModified)
    } else {
      res.WriteHeader(http.StatusOK)
      _, err := res.Write(data)
      if err != nil {
        panic(err)
      }
    }
  })
  println("Server running on 127.0.0.1:3000")
  http.ListenAndServe(":3000", nil)
}
```
https://pkg.go.dev/github.com/pyros2097/go-embed

 

The `File` struct would have to be our own and not the std's, because that one is backed by a real file on a file system. It would also make sense to rename it to something like `EmbeddedFile`.
I've started working on this and should have a PR up soon.
> The File struct would have to be our own and not the std's, because that one is backed by a real file on a file system. It would also make sense to rename it to something like EmbeddedFile.

Yep exactly. Thought I should have made it more clear. `EmbeddedFile` makes more sense. 

Thanks.


Fix patch:
diff --git a/Cargo.toml b/Cargo.toml
index a14fe49..1787ffe 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -43,6 +43,9 @@ tokio = { version = "0.2", optional = true, features = ["macros"] }
 warp = { version = "0.2", default-features = false, optional = true }
 rocket = { version = "0.4.5", default-features = false, optional = true }
 
+[dev-dependencies]
+sha2 = "0.9"
+
 [features]
 debug-embed = ["rust-embed-impl/debug-embed", "rust-embed-utils/debug-embed"]
 interpolate-folder-path = ["rust-embed-impl/interpolate-folder-path"]
diff --git a/examples/actix.rs b/examples/actix.rs
index 3d3e890..047448b 100644
--- a/examples/actix.rs
+++ b/examples/actix.rs
@@ -12,7 +12,7 @@ struct Asset;
 fn handle_embedded_file(path: &str) -> HttpResponse {
   match Asset::get(path) {
     Some(content) => {
-      let body: Body = match content {
+      let body: Body = match content.data {
         Cow::Borrowed(bytes) => bytes.into(),
         Cow::Owned(bytes) => bytes.into(),
       };
diff --git a/examples/basic.rs b/examples/basic.rs
index e2f0cdc..b143e61 100644
--- a/examples/basic.rs
+++ b/examples/basic.rs
@@ -6,5 +6,5 @@ struct Asset;
 
 fn main() {
   let index_html = Asset::get("index.html").unwrap();
-  println!("{:?}", std::str::from_utf8(index_html.as_ref()));
+  println!("{:?}", std::str::from_utf8(index_html.data.as_ref()));
 }
diff --git a/examples/rocket.rs b/examples/rocket.rs
index 9b3ea67..4aa6169 100644
--- a/examples/rocket.rs
+++ b/examples/rocket.rs
@@ -18,7 +18,7 @@ struct Asset;
 fn index<'r>() -> response::Result<'r> {
   Asset::get("index.html").map_or_else(
     || Err(Status::NotFound),
-    |d| response::Response::build().header(ContentType::HTML).sized_body(Cursor::new(d)).ok(),
+    |d| response::Response::build().header(ContentType::HTML).sized_body(Cursor::new(d.data)).ok(),
   )
 }
 
@@ -34,7 +34,7 @@ fn dist<'r>(file: PathBuf) -> response::Result<'r> {
         .and_then(OsStr::to_str)
         .ok_or_else(|| Status::new(400, "Could not get file extension"))?;
       let content_type = ContentType::from_extension(ext).ok_or_else(|| Status::new(400, "Could not get file content type"))?;
-      response::Response::build().header(content_type).sized_body(Cursor::new(d)).ok()
+      response::Response::build().header(content_type).sized_body(Cursor::new(d.data)).ok()
     },
   )
 }
diff --git a/examples/warp.rs b/examples/warp.rs
index 4cea5fb..1164790 100644
--- a/examples/warp.rs
+++ b/examples/warp.rs
@@ -26,7 +26,7 @@ fn serve_impl(path: &str) -> Result<impl Reply, Rejection> {
   let asset = Asset::get(path).ok_or_else(warp::reject::not_found)?;
   let mime = mime_guess::from_path(path).first_or_octet_stream();
 
-  let mut res = Response::new(asset.into());
+  let mut res = Response::new(asset.data.into());
   res.headers_mut().insert("content-type", HeaderValue::from_str(mime.as_ref()).unwrap());
   Ok(res)
 }
diff --git a/impl/src/lib.rs b/impl/src/lib.rs
index ab97dee..94c0846 100644
--- a/impl/src/lib.rs
+++ b/impl/src/lib.rs
@@ -46,7 +46,7 @@ fn embedded(ident: &syn::Ident, folder_path: String, prefix: Option<&str>) -> To
   quote! {
       #not_debug_attr
       impl #ident {
-          pub fn get(file_path: &str) -> Option<std::borrow::Cow<'static, [u8]>> {
+          pub fn get(file_path: &str) -> Option<rust_embed::EmbeddedFile> {
             #handle_prefix
             match file_path.replace("\\", "/").as_str() {
                 #(#match_values)*
@@ -66,7 +66,7 @@ fn embedded(ident: &syn::Ident, folder_path: String, prefix: Option<&str>) -> To
 
       #not_debug_attr
       impl rust_embed::RustEmbed for #ident {
-        fn get(file_path: &str) -> Option<std::borrow::Cow<'static, [u8]>> {
+        fn get(file_path: &str) -> Option<rust_embed::EmbeddedFile> {
           #ident::get(file_path)
         }
         fn iter() -> rust_embed::Filenames {
@@ -89,19 +89,11 @@ fn dynamic(ident: &syn::Ident, folder_path: String, prefix: Option<&str>) -> Tok
   quote! {
       #[cfg(debug_assertions)]
       impl #ident {
-          pub fn get(file_path: &str) -> Option<std::borrow::Cow<'static, [u8]>> {
-              use std::fs;
-              use std::path::Path;
-
+          pub fn get(file_path: &str) -> Option<rust_embed::EmbeddedFile> {
               #handle_prefix
 
-              let file_path = Path::new(#folder_path).join(file_path.replace("\\", "/"));
-              match fs::read(file_path) {
-                  Ok(contents) => Some(std::borrow::Cow::from(contents)),
-                  Err(_e) =>  {
-                      return None
-                  }
-              }
+              let file_path = std::path::Path::new(#folder_path).join(file_path.replace("\\", "/"));
+              rust_embed::utils::read_file_from_fs(&file_path).ok()
           }
 
           pub fn iter() -> impl Iterator<Item = std::borrow::Cow<'static, str>> {
@@ -113,7 +105,7 @@ fn dynamic(ident: &syn::Ident, folder_path: String, prefix: Option<&str>) -> Tok
 
       #[cfg(debug_assertions)]
       impl rust_embed::RustEmbed for #ident {
-        fn get(file_path: &str) -> Option<std::borrow::Cow<'static, [u8]>> {
+        fn get(file_path: &str) -> Option<rust_embed::EmbeddedFile> {
           #ident::get(file_path)
         }
         fn iter() -> rust_embed::Filenames {
@@ -138,25 +130,34 @@ fn generate_assets(ident: &syn::Ident, folder_path: String, prefix: Option<Strin
   }
 }
 
-#[cfg(not(feature = "compression"))]
 fn embed_file(rel_path: &str, full_canonical_path: &str) -> TokenStream2 {
-  quote! {
-    #rel_path => {
-        let bytes = &include_bytes!(#full_canonical_path)[..];
-        Some(std::borrow::Cow::from(bytes))
-    },
-  }
-}
+  let file = rust_embed_utils::read_file_from_fs(Path::new(full_canonical_path)).expect("File should be readable");
+  let hash = file.metadata.sha256_hash();
+  let last_modified = match file.metadata.last_modified() {
+    Some(last_modified) => quote! { Some(#last_modified) },
+    None => quote! { None },
+  };
+
+  let embedding_code = if cfg!(feature = "compression") {
+    quote! {
+      rust_embed::flate!(static FILE: [u8] from #full_canonical_path);
+      let bytes = &FILE[..];
+    }
+  } else {
+    quote! {
+      let bytes = &include_bytes!(#full_canonical_path)[..];
+    }
+  };
 
-#[cfg(feature = "compression")]
-fn embed_file(rel_path: &str, full_canonical_path: &str) -> TokenStream2 {
   quote! {
-    #rel_path => {
-        rust_embed::flate!(static FILE: [u8] from #full_canonical_path);
+      #rel_path => {
+          #embedding_code
 
-        let bytes = &FILE[..];
-        Some(std::borrow::Cow::from(bytes))
-    },
+          Some(rust_embed::EmbeddedFile {
+              data: std::borrow::Cow::from(bytes),
+              metadata: rust_embed::Metadata::__rust_embed_new([#(#hash),*], #last_modified)
+          })
+      }
   }
 }
 
@@ -213,7 +214,7 @@ fn impl_rust_embed(ast: &syn::DeriveInput) -> TokenStream2 {
                   when the `interpolate-folder-path` feature is enabled.";
     }
 
-    panic!(message);
+    panic!("{}", message);
   };
 
   generate_assets(&ast.ident, folder_path, prefix)
diff --git a/src/lib.rs b/src/lib.rs
index e1110f6..59f4899 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -8,8 +8,9 @@ pub use include_flate::flate;
 extern crate rust_embed_impl;
 pub use rust_embed_impl::*;
 
+pub use rust_embed_utils::{EmbeddedFile, Metadata};
+
 #[doc(hidden)]
-#[cfg(all(debug_assertions, not(feature = "debug-embed")))]
 pub extern crate rust_embed_utils as utils;
 
 /// A directory of binary assets.
@@ -37,7 +38,7 @@ pub trait RustEmbed {
   ///
   /// Otherwise, the bytes are read from the file system on each call and a
   /// `Cow::Owned(Vec<u8>)` is returned.
-  fn get(file_path: &str) -> Option<std::borrow::Cow<'static, [u8]>>;
+  fn get(file_path: &str) -> Option<EmbeddedFile>;
 
   /// Iterates the files in this assets folder.
   ///
diff --git a/utils/Cargo.toml b/utils/Cargo.toml
index e33cfc3..c73ec9b 100644
--- a/utils/Cargo.toml
+++ b/utils/Cargo.toml
@@ -13,6 +13,7 @@ edition = "2018"
 
 [dependencies]
 walkdir = "2.3.1"
+sha2 = "0.9"
 
 [features]
 debug-embed = []
diff --git a/utils/src/lib.rs b/utils/src/lib.rs
index 81ff3d2..841ac75 100644
--- a/utils/src/lib.rs
+++ b/utils/src/lib.rs
@@ -1,4 +1,11 @@
 #![forbid(unsafe_code)]
+
+use sha2::Digest;
+use std::borrow::Cow;
+use std::path::Path;
+use std::time::SystemTime;
+use std::{fs, io};
+
 #[cfg_attr(all(debug_assertions, not(feature = "debug-embed")), allow(unused))]
 pub struct FileEntry {
   pub rel_path: String,
@@ -26,6 +33,57 @@ pub fn get_files(folder_path: String) -> impl Iterator<Item = FileEntry> {
     })
 }
 
+/// A file embedded into the binary
+pub struct EmbeddedFile {
+  pub data: Cow<'static, [u8]>,
+  pub metadata: Metadata,
+}
+
+/// Metadata about an embedded file
+pub struct Metadata {
+  hash: [u8; 32],
+  last_modified: Option<u64>,
+}
+
+impl Metadata {
+  #[doc(hidden)]
+  pub fn __rust_embed_new(hash: [u8; 32], last_modified: Option<u64>) -> Self {
+    Self { hash, last_modified }
+  }
+
+  /// The SHA256 hash of the file
+  pub fn sha256_hash(&self) -> [u8; 32] {
+    self.hash
+  }
+
+  /// The last modified date in seconds since the UNIX epoch. If the underlying
+  /// platform/file-system does not support this, None is returned.
+  pub fn last_modified(&self) -> Option<u64> {
+    self.last_modified
+  }
+}
+
+pub fn read_file_from_fs(file_path: &Path) -> io::Result<EmbeddedFile> {
+  let data = fs::read(file_path)?;
+  let data = Cow::from(data);
+
+  let mut hasher = sha2::Sha256::new();
+  hasher.update(&data);
+  let hash: [u8; 32] = hasher.finalize().into();
+
+  let last_modified = fs::metadata(file_path)?.modified().ok().map(|last_modified| {
+    last_modified
+      .duration_since(SystemTime::UNIX_EPOCH)
+      .expect("Time before the UNIX epoch is unsupported")
+      .as_secs()
+  });
+
+  Ok(EmbeddedFile {
+    data,
+    metadata: Metadata { hash, last_modified },
+  })
+}
+
 fn path_to_str<P: AsRef<std::path::Path>>(p: P) -> String {
   p.as_ref().to_str().expect("Path does not have a string representation").to_owned()
 }


Test patch:
diff --git a/tests/interpolated_path.rs b/tests/interpolated_path.rs
index 3534164..0df2366 100644
--- a/tests/interpolated_path.rs
+++ b/tests/interpolated_path.rs
@@ -7,15 +7,9 @@ struct Asset;
 
 #[test]
 fn get_works() {
-  if Asset::get("index.html").is_none() {
-    panic!("index.html should exist");
-  }
-  if Asset::get("gg.html").is_some() {
-    panic!("gg.html should not exist");
-  }
-  if Asset::get("images/llama.png").is_none() {
-    panic!("llama.png should exist");
-  }
+  assert!(Asset::get("index.html").is_some(), "index.html should exist");
+  assert!(Asset::get("gg.html").is_none(), "gg.html should not exist");
+  assert!(Asset::get("images/llama.png").is_some(), "llama.png should exist");
 }
 
 #[test]
diff --git a/tests/lib.rs b/tests/lib.rs
index 729ca99..a4636aa 100644
--- a/tests/lib.rs
+++ b/tests/lib.rs
@@ -7,15 +7,9 @@ struct Asset;
 
 #[test]
 fn get_works() {
-  if Asset::get("index.html").is_none() {
-    panic!("index.html should exist");
-  }
-  if Asset::get("gg.html").is_some() {
-    panic!("gg.html should not exist");
-  }
-  if Asset::get("images/llama.png").is_none() {
-    panic!("llama.png should exist");
-  }
+  assert!(Asset::get("index.html").is_some(), "index.html should exist");
+  assert!(Asset::get("gg.html").is_none(), "gg.html should not exist");
+  assert!(Asset::get("images/llama.png").is_some(), "llama.png should exist");
 }
 
 /// Using Windows-style path separators (`\`) is acceptable
diff --git a/tests/metadata.rs b/tests/metadata.rs
new file mode 100644
index 0000000..2e755a6
--- /dev/null
+++ b/tests/metadata.rs
@@ -0,0 +1,24 @@
+use rust_embed::{EmbeddedFile, RustEmbed};
+use sha2::Digest;
+
+#[derive(RustEmbed)]
+#[folder = "examples/public/"]
+struct Asset;
+
+#[test]
+fn hash_is_accurate() {
+  let index_file: EmbeddedFile = Asset::get("index.html").expect("index.html exists");
+  let mut hasher = sha2::Sha256::new();
+  hasher.update(index_file.data);
+  let expected_hash: [u8; 32] = hasher.finalize().into();
+
+  assert_eq!(index_file.metadata.sha256_hash(), expected_hash);
+}
+
+#[test]
+fn last_modified_is_accurate() {
+  let index_file: EmbeddedFile = Asset::get("index.html").expect("index.html exists");
+  let expected_datetime_utc = 1527818165;
+
+  assert_eq!(index_file.metadata.last_modified(), Some(expected_datetime_utc));
+}