Repository: gakonst/ethers-rs
Pull request #994 (2022-03-07 06:52:18)

feat(solc): yul compilation
## Motivation

Currently Foundry and ethers-rs is not capable of compiling Yul sources (https://github.com/gakonst/foundry/issues/759)

## Solution

Add yul compilation

ControlCplusControlV: > Yeah I was definitely thinking about adding abi for yul. My thinking was to load file with the same name as abi. E.g. if you have contracts/tokens/ERC20.yul then abi would be loaded from contracts/tokens/ERC20.abi.json. But that'd a good next step. 
> In any case I'm ok to merge this one or yours or some combined version. The key is to have a mix of Solidity / Yul sources in one project which will be kinda cool since hardhat and other guys don't have it :)

I can move over some of my changes here (mostly with the test) as suggestions and you can move them in as needed. 

The mix is definitely really cool as only Truffle can do it. What would you think of an approach where we have a way to cast a Solidity interface over the Yul file (so yul_file_name.abi.sol) and compile the interface, and put the ABI on top of the Yul artifact, then those who want to skip it with call() have the option to, while others can use it like nothing is different
ControlCplusControlV: @gakonst yeah will make that PR later today as I am going to add in that change to handle Yul ABI alongside with it.
ControlCplusControlV: My PR is up to @alexeuler's branch, still needs some work with one last type resolution issue which I can hopefully get resolved tomorrow as ABI should be fully functional from there! 
ControlCplusControlV: So type issues is fixed, but need help with one last thing. In this part of my fork of [artifact_output mod.rs](https://github.com/ControlCplusControlV/ethers-rs-1/blob/feature/yul_compilation/ethers-solc/src/artifact_output/mod.rs#L580) I have both the abi I need to replace and the artifact of which I need to insert the ABI into. But it doesn't appear there is a clean way to do that as Artifact won't let me modify the ABI directly and if I convert it into a CompactContract I am running into issues converting it back, so was curious as to the best way to do this or any other approaches. Will continue testing things but wanted to ask here in case there is something I am missing which could speed things up
mattsse: I see, what modifications are necessary?
it seems like the proper way is to extend the `Artifact` trait some functions specifically for yul?
ControlCplusControlV: The ABI needs to be accessible so that it can be replaced for a given `Artifact` or `ArtifactFile`. I will try extending artifact although there is a lot of unwrapping needed to get to the ABI and with all the types that implement Artifact I was hoping there would be another way which requires less refactoring
ControlCplusControlV: @mattsse is there a specific way you envision extending `Artifact` ? As I need to replace the ABI but I haven't found a good way to access the ABI directly, as all the functions involve converting to a CompactContract to fetch values like ABI or Bytecode, but as there is no way to convert it back I haven't been able to get that approach to work. Is there a better way to directly access that value? 
mattsse: > @mattsse is there a specific way you envision extending `Artifact` ? As I need to replace the ABI but I haven't found a good way to access the ABI directly, as all the functions involve converting to a CompactContract to fetch values like ABI or Bytecode, but as there is no way to convert it back I haven't been able to get that approach to work. Is there a better way to directly access that value?

you mean the `ethers::abi::Abi` type? that type is baked into everything. 
What kind of modification/conversion do need need?

I don't have the full picture yet, and would need some more context on what needs to be done here https://github.com/ControlCplusControlV/ethers-rs-1/blob/feature/yul_compilation/ethers-solc/src/artifact_output/mod.rs#L580
ControlCplusControlV: @mattsse I was actually able to resolve it by avoiding type conversions and just saving the Contract Object for each Yul contract, so that I could then just modify and replace it later. Just have one bug where it is compiling correctly but it seems the caching is overwriting the file contents instead of a cache, if you want to help I have pushed all my code to my fork and the test is fairly reliable producing it (It overwrites SimpleStore.yul with the cache of the artifact)
ControlCplusControlV: @gakonst (and @alexeuler as it) could you review my changes for ABI to start moving it upstream? PR is [here](https://github.com/alexeuler/ethers-rs/pull/1) and my fork is [here](https://github.com/ControlCplusControlV/ethers-rs-1). So sorry for how long this took, did realize how bad my rust type skills are (although they've improved much more now!). I've added tests but as I was iterating so much code style is a bit sloppy so if there are any code style choices I can gladly implement them. 

It is a bit polluted with commits so squashing and merging will be much cleaner.
gakonst: My candid recommendation would be that we merge this as-is, and @ControlCplusControlV can post a separate PR on top of master, given that it's not a blocker for this PR

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c44561a0..a9002f81 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -47,6 +47,7 @@
 - Add a getter to `ProjectCompileOutput` that returns a mapping of compiler
   versions to a vector of name + contract struct tuples
   [#908](https://github.com/gakonst/ethers-rs/pull/908)
+- Add Yul compilation [994](https://github.com/gakonst/ethers-rs/pull/994)
 - Enforce commutativity of ENS reverse resolution
   [#996](https://github.com/gakonst/ethers-rs/pull/996)
 
diff --git a/ethers-solc/src/artifacts/mod.rs b/ethers-solc/src/artifacts/mod.rs
index 0d5038a4..0487ba48 100644
--- a/ethers-solc/src/artifacts/mod.rs
+++ b/ethers-solc/src/artifacts/mod.rs
@@ -52,13 +52,37 @@ pub struct CompilerInput {
 
 impl CompilerInput {
     /// Reads all contracts found under the path
-    pub fn new(path: impl AsRef<Path>) -> Result<Self, SolcIoError> {
+    pub fn new(path: impl AsRef<Path>) -> Result<Vec<Self>, SolcIoError> {
         Source::read_all_from(path.as_ref()).map(Self::with_sources)
     }
 
     /// Creates a new Compiler input with default settings and the given sources
-    pub fn with_sources(sources: Sources) -> Self {
-        Self { language: "Solidity".to_string(), sources, settings: Default::default() }
+    pub fn with_sources(sources: Sources) -> Vec<Self> {
+        let mut solidity_sources = BTreeMap::new();
+        let mut yul_sources = BTreeMap::new();
+        for (path, source) in sources {
+            if path.extension() == Some(std::ffi::OsStr::new("yul")) {
+                yul_sources.insert(path, source);
+            } else {
+                solidity_sources.insert(path, source);
+            }
+        }
+        let mut res = Vec::new();
+        if !solidity_sources.is_empty() {
+            res.push(Self {
+                language: "Solidity".to_string(),
+                sources: solidity_sources,
+                settings: Default::default(),
+            });
+        }
+        if !yul_sources.is_empty() {
+            res.push(Self {
+                language: "Yul".to_string(),
+                sources: yul_sources,
+                settings: Default::default(),
+            });
+        }
+        res
     }
 
     /// Sets the settings for compilation
@@ -99,12 +123,6 @@ impl CompilerInput {
     }
 }
 
-impl Default for CompilerInput {
-    fn default() -> Self {
-        Self::with_sources(Default::default())
-    }
-}
-
 #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
 #[serde(rename_all = "camelCase")]
 pub struct Settings {
@@ -805,6 +823,12 @@ impl CompilerOutput {
             err.source_location.as_ref().map(|s| files.contains(s.file.as_str())).unwrap_or(true)
         });
     }
+
+    pub fn merge(&mut self, other: CompilerOutput) {
+        self.errors.extend(other.errors);
+        self.contracts.extend(other.contracts);
+        self.sources.extend(other.sources);
+    }
 }
 
 /// A wrapper helper type for the `Contracts` type alias
diff --git a/ethers-solc/src/compile/mod.rs b/ethers-solc/src/compile/mod.rs
index 9841fcc6..7f4d48db 100644
--- a/ethers-solc/src/compile/mod.rs
+++ b/ethers-solc/src/compile/mod.rs
@@ -410,7 +410,12 @@ impl Solc {
     /// Convenience function for compiling all sources under the given path
     pub fn compile_source(&self, path: impl AsRef<Path>) -> Result<CompilerOutput> {
         let path = path.as_ref();
-        self.compile(&CompilerInput::new(path)?)
+        let mut res: CompilerOutput = Default::default();
+        for input in CompilerInput::new(path)? {
+            let output = self.compile(&input)?;
+            res.merge(output)
+        }
+        Ok(res)
     }
 
     /// Same as [`Self::compile()`], but only returns those files which are included in the
@@ -425,7 +430,7 @@ impl Solc {
     /// # fn main() -> Result<(), Box<dyn std::error::Error>> {
     ///  use ethers_solc::{CompilerInput, Solc};
     /// let solc = Solc::default();
-    /// let input = CompilerInput::new("./contracts")?;
+    /// let input = CompilerInput::new("./contracts")?[0].clone();
     /// let output = solc.compile_exact(&input)?;
     /// # Ok(())
     /// # }
@@ -571,8 +576,8 @@ impl Solc {
     /// use ethers_solc::{CompilerInput, Solc};
     /// let solc1 = Solc::default();
     /// let solc2 = Solc::default();
-    /// let input1 = CompilerInput::new("contracts").unwrap();
-    /// let input2 = CompilerInput::new("src").unwrap();
+    /// let input1 = CompilerInput::new("contracts").unwrap()[0].clone();
+    /// let input2 = CompilerInput::new("src").unwrap()[0].clone();
     ///
     /// let outputs = Solc::compile_many([(solc1, input1), (solc2, input2)], 2).await.flattened().unwrap();
     /// # }
diff --git a/ethers-solc/src/compile/project.rs b/ethers-solc/src/compile/project.rs
index 91ba7d5c..19e36a3c 100644
--- a/ethers-solc/src/compile/project.rs
+++ b/ethers-solc/src/compile/project.rs
@@ -341,24 +341,23 @@ fn compile_sequential(
             solc.args
         );
 
-        let input = CompilerInput::with_sources(sources)
-            .settings(settings.clone())
-            .normalize_evm_version(&version)
-            .with_remappings(paths.remappings.clone());
-
-        tracing::trace!(
-            "calling solc `{}` with {} sources {:?}",
-            version,
-            input.sources.len(),
-            input.sources.keys()
-        );
-
-        report::solc_spawn(&solc, &version, &input);
-        let output = solc.compile_exact(&input)?;
-        report::solc_success(&solc, &version, &output);
-        tracing::trace!("compiled input, output has error: {}", output.has_error());
-
-        aggregated.extend(version, output);
+        for input in CompilerInput::with_sources(sources) {
+            let input = input
+                .settings(settings.clone())
+                .normalize_evm_version(&version)
+                .with_remappings(paths.remappings.clone());
+            tracing::trace!(
+                "calling solc `{}` with {} sources {:?}",
+                version,
+                input.sources.len(),
+                input.sources.keys()
+            );
+            report::solc_spawn(&solc, &version, &input);
+            let output = solc.compile_exact(&input)?;
+            report::solc_success(&solc, &version, &output);
+            tracing::trace!("compiled input, output has error: {}", output.has_error());
+            aggregated.extend(version.clone(), output);
+        }
     }
     Ok(aggregated)
 }
@@ -383,13 +382,14 @@ fn compile_parallel(
             // nothing to compile
             continue
         }
+        for input in CompilerInput::with_sources(sources) {
+            let job = input
+                .settings(settings.clone())
+                .normalize_evm_version(&version)
+                .with_remappings(paths.remappings.clone());
 
-        let job = CompilerInput::with_sources(sources)
-            .settings(settings.clone())
-            .normalize_evm_version(&version)
-            .with_remappings(paths.remappings.clone());
-
-        jobs.push((solc, version, job))
+            jobs.push((solc.clone(), version.clone(), job))
+        }
     }
 
     // start a rayon threadpool that will execute all `Solc::compile()` processes
diff --git a/ethers-solc/src/utils.rs b/ethers-solc/src/utils.rs
index c15a2697..b269d52f 100644
--- a/ethers-solc/src/utils.rs
+++ b/ethers-solc/src/utils.rs
@@ -67,7 +67,9 @@ pub fn source_files(root: impl AsRef<Path>) -> Vec<PathBuf> {
         .into_iter()
         .filter_map(Result::ok)
         .filter(|e| e.file_type().is_file())
-        .filter(|e| e.path().extension().map(|ext| ext == "sol").unwrap_or_default())
+        .filter(|e| {
+            e.path().extension().map(|ext| (ext == "sol") || (ext == "yul")).unwrap_or_default()
+        })
         .map(|e| e.path().into())
         .collect()
 }


Test patch:
diff --git a/ethers-solc/test-data/yul-sample/Dapp.sol b/ethers-solc/test-data/yul-sample/Dapp.sol
new file mode 100644
index 00000000..762db2c3
--- /dev/null
+++ b/ethers-solc/test-data/yul-sample/Dapp.sol
@@ -0,0 +1,7 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+pragma solidity >=0.6.6;
+
+contract Dapp {
+
+    function modified() public {}
+}
diff --git a/ethers-solc/test-data/yul-sample/SimpleStore.yul b/ethers-solc/test-data/yul-sample/SimpleStore.yul
new file mode 100644
index 00000000..aba14daa
--- /dev/null
+++ b/ethers-solc/test-data/yul-sample/SimpleStore.yul
@@ -0,0 +1,11 @@
+object "SimpleStore" {
+  code {
+    datacopy(0, dataoffset("Runtime"), datasize("Runtime"))
+    return(0, datasize("Runtime"))
+  }
+  object "Runtime" {
+    code {
+      calldatacopy(0, 0, 36) // write calldata to memory
+    }
+  }
+}
diff --git a/ethers-solc/tests/project.rs b/ethers-solc/tests/project.rs
index 3129da7c..c1e7ee28 100644
--- a/ethers-solc/tests/project.rs
+++ b/ethers-solc/tests/project.rs
@@ -92,6 +92,36 @@ fn can_compile_dapp_sample() {
     assert_eq!(cache, updated_cache);
 }
 
+#[test]
+fn can_compile_yul_sample() {
+    let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("test-data/yul-sample");
+    let paths = ProjectPathsConfig::builder().sources(root);
+    let project = TempProject::<ConfigurableArtifacts>::new(paths).unwrap();
+
+    let compiled = project.compile().unwrap();
+    assert!(compiled.find("Dapp").is_some());
+    assert!(compiled.find("SimpleStore").is_some());
+    assert!(!compiled.has_compiler_errors());
+
+    // nothing to compile
+    let compiled = project.compile().unwrap();
+    assert!(compiled.find("Dapp").is_some());
+    assert!(compiled.find("SimpleStore").is_some());
+    assert!(compiled.is_unchanged());
+
+    let cache = SolFilesCache::read(project.cache_path()).unwrap();
+
+    // delete artifacts
+    std::fs::remove_dir_all(&project.paths().artifacts).unwrap();
+    let compiled = project.compile().unwrap();
+    assert!(compiled.find("Dapp").is_some());
+    assert!(compiled.find("SimpleStore").is_some());
+    assert!(!compiled.is_unchanged());
+
+    let updated_cache = SolFilesCache::read(project.cache_path()).unwrap();
+    assert_eq!(cache, updated_cache);
+}
+
 #[test]
 fn can_compile_configured() {
     let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("test-data/dapp-sample");