Title: ethPM_V3_Specification_Working_Group
Category: Magicians

Author: njgheorghita
Date: 2020-03-04T21:08:04.132Z

This is an effort to consolidate various datatype specifications (ethpm, solidity & vyper compiler metadata, input/output json) into a single specification to facilitate interoperability and leverage network effects.
The current V3 spec should be considered a rough draft, so now is the time for proposing any changes. Once we reach general agreement on the spec in this forum, I will update the official human & machine readable specs to reflect the changes.
The timeline for this project is to have a well-defined, agreed upon specification before the Solidity Summit in Berlin (April 29). At least 3 of us (@chriseth, @gnidan, @njgheorghita) plan to be at the Summit, to discuss any final details before freezing the spec. The more the merrier if you’re able to make it to Berlin!
Thanks for all your input and thoughts in advance! There is a range of different interests coalescing here, and it’s exciting to imagine the near-future where we all benefit from using a compatible spec.
Link for the ethPM V3 rough draft can be found here. Let’s keep the conversation happening here on ethmagicians, and I’ll make any updates to the spec as necessary.

@jpitts: Adding more context to this discussion:
Registry: http://explorer.ethpm.com/
Website: http://www.ethpm.com/
v2 Spec repo: https://github.com/ethpm/ethpm-spec/
Gitter Channel: https://gitter.im/ethpm/Lobby

Author: rumkin
Date: 2020-03-06T18:44:45.392Z

Using version for manifest version is confusing for me, it’s totally not what I’d expect version to mean. Also this makes package version to be called package_version and thus the name of a package to be  package_name. I think it’s better to call it manifest and remove package_ prefixes at all. This would make field names clear and obvious:
{
  "manifest": "ethpm/3.0",
  "name": "package",
  "version": "1.0.0"
}

This is what I found after a quick look.

Author: fubuloubu
Date: 2020-03-06T19:35:35.988Z

Agreed, but it may be impractical to change at this point because V1 sort of set these fields in stone (particularly version) so you could parse out what version the manifest is encoded in

Author: rumkin
Date: 2020-03-06T20:00:43.613Z

Yep, but before Ethereum 2.0, it’s ok to break things. In my opinion this time is some kind of buffer to experiment,  when you can hardfork yourself. And technologies here should be made user friendly and slick like Ruby IDK, not like C. It’s important because Ethereum 2.0 would be a long-time player and it’s better to invest in this period of technology life. Just not a time to create a legacy.

Author: fubuloubu
Date: 2020-03-06T20:07:40.028Z

I don’t think this standard would be related to ETH 2.0, it sort of sits by itself in relation to current tooling. It has some adoption, so I am unsure if there would be appetite for a breaking change like this, but I could be wrong.

Author: rumkin
Date: 2020-03-06T20:40:55.044Z

I’ve checked the doc again, and it’s actually contains breaking changes. One of them is renaming manifest_version into version and version into package_version, which I’m against. So it’s senseless to discuss.
@njgheorghita Also I’d suggest to use camel case instead of underscores in JSON. While underscores has better readability, camel case is de facto standard for JSON, and solidity compiler supports this. I think that host technology style should be chosen, just couldn’t find a good reason to have both of them.

Author: njgheorghita
Date: 2020-03-06T21:36:00.061Z

@rumkin Yup, I’ve gotta say I do prefer what you’ve proposed, thanks for submitting! Personally, I’ve never been a fan of the old naming conventions, they could be better. We do have breaking changes in this spec update, and all tooling will have to update to satisfy those changes, so now is a great time to squeeze in any breaking improvements. (afaik that responsibility lies mostly b/w myself, brownie (@iamdefinitelyahuman) , truffle, and ethpm-rs (@fubuloubu) if that’s still ongoing).
The primary reason I opted for the proposed changes (version, package_name, package_version) is because the solidity compiler currently supports the version keyword to define the metadata specification version - which maps nicely to what used to be the manifest_version field. I’d like to hear from @chriseth his thoughts? And whether the changes proposed by @rumkin could be easily adopted by the compiler.
In terms of camelCase vs snake_case - it’s a more significant breaking change. I’m open to the change if there is general consensus that it’s important / worthwhile.

Author: gh1dra
Date: 2020-03-11T18:44:07.223Z

Is there a particular reasoning for why the devdoc and userdoc are being surfaced for a particular contract inside contract_types? From my understanding of the spec, those fields be made available if the CompilerInformationObject field is already given. Is the idea to give users the choice about what kind of information they choose to provide about the compilation?

Author: chriseth
Date: 2020-03-12T13:01:31.505Z

I’m fine with breaking Solidity metadata with regards to “version” if it is a dealbreaker for EthPM as long as it is still possible to easily determine whether the given json is a v1 Solidity metadata json.
I would also prefer camelCase.
Funther remarks (I’m really not sure if this forum-style discussion is the best way to do this):
sources: " Paths must resolve to a path within the current virtual directory." - do we really need this requirement? What does it mean for a path to resolve to such a path? How do you determine if it resolves to a directory or to a file?
“contract_types”:
If there are multiple contract types specified - is there a way to provide the “main” contract in some way? For Solidity metadata, there is exactly one contract type that is the one currently being compiled. Is the expectation that in this case, there is only one item in this object?
contract alias: In the Solidity compiler, we identify names by prefixing the file name (including path) followed by a “:”. How do contract aliases identify the actual contract in the source?
Why is the compiler a sub-field of the contract type? I see that different compilers could be used for different contract, but in that case, shouldn’t the sources also be a sub-field of the contract type?

Author: rumkin
Date: 2020-03-21T12:28:54.899Z

I’m really not sure if this forum-style discussion is the best way to do this

What’s the best place to do so? Let’s move it there. It’s possible to separate some questions into issues, but IDK where to put them.

If there are multiple contract types specified - is there a way to provide the “main” contract in some way?

I think it’s reasonable to define default keyword in Solidity to specify it explicitly. It would help to export things with automated tools. Libraries authors usually define single contract per file, but it’s only an agreement and it’s not certain for all libraries. Export such a contract as the main one doesn’t seem correct due to this uncertainty.

Author: njgheorghita
Date: 2020-03-24T16:14:19.423Z

Is the idea to give users the choice about what kind of information they choose to provide about the compilation?

@gh1dra The compiler field is for information about the compiler and its settings that were used to generate the outputs (abi, bytecodes, userdoc, devdoc) rather than containing the outputs of the compilation. As you said, this gives the user more flexibility in choosing what fields they want or are able to include in a manifest.

I’m fine with breaking Solidity metadata with regards to “version” if it is a dealbreaker for EthPM

It’s not a dealbreaker for me, but I do prefer the scheme ("manifest", "name", "version") that @rumkin proposed the best. It’s the cleanest / easiest to understand.

as long as it is still possible to easily determine whether the given json is a v1 Solidity metadata json.

If we adopt this scheme, I’m not so sure this will be easily achieved. As @fubuloubu pointed out, there will be ethpm v1 & v2 manifests that have a "version" field of "1" (referring to the version of that specific package release, rather than the specification it conforms to). These will be impossible to simply distinguish from v1 Solidity metadata json (which also have a "version" field of "1"). Unless the tooling has the knowledge to search for a "manifest" field (required in all ethpm v3 manifests), validate that it is "3" - and then it will be able to distinguish b/w the two specifications. So it is possible reliably determine which specification a json object is, but it won’t be trivial. If this is an acceptable amount of complexity for @chriseth, I’ll update the spec to reflect this change.

I would also prefer camelCase.

 I’m also ok with this change. Unless any of the python/vyper ppl have strong objections, I’ll update the spec.

sources: " Paths  must  resolve to a path within the current virtual directory." - do we really need this requirement? What does it mean for a path to resolve to such a path?

For ethpm, we need the ability to faithfully recreate the original source tree. Otherwise, when users “install” a package to disk - if the paths are not the same paths of the original source tree, then they might not be able to re-compile the sources.
In ethpm v1/v2 we required all paths to be relative paths that begin with ./ (the root of the virtual directory), which has worked well. Though, I understand the solidity compiler might not want to enforce such a requirement. If we relax this requirement, we’ll need to define an additional mechanism (like a filesystem key) that maps the paths used to valid filesystem paths - which will be required for packages to have the capability to be written to disk (“installed”).

How do you determine if it resolves to a directory or to a file?

If the path points to a content-addressed uri, the uri must be resolved to determine whether it’s a directory or a file. The only other option is for the path to point to an inlined string of the source contract which would always be a single file.

If there are multiple contract types specified - is there a way to provide the “main” contract in some way? For Solidity metadata, there is exactly one contract type that is the one currently being compiled. Is the expectation that in this case, there is only one item in this object?

Yup, that’s how I’d see it working out.

contract alias: In the Solidity compiler, we identify names by prefixing the file name (including path) followed by a “:”. How do contract aliases identify the actual contract in the source?

In ethpm a contract alias is defined as <contract-name>[<identifier>]. In ethpm v1/v2 there was no mechanism to map a contract type to a source - however in v3 it seems to me as though the compilationTarget field in the compiler settings would serve this function.

Why is the compiler a sub-field of the contract type? I see that different compilers could be used for different contract, but in that case, shouldn’t the sources also be a sub-field of the contract type?

I’m not sure I understand this concern exactly. There are packages that would be useful with just "sources" and no "contract_types". Also, different compilers could be used to generate different "contract_types" for the same source. This would introduce a lot of redundancy if sources were located in "contract_types".

I’m really not sure if this forum-style discussion is the best way to do this

Agreed, I’m starting to think that it’s time to move this to github, where we can have more nuanced threads around specific concerns. I’ll work on migrating the changes discussed here to the ethpm-spec repository - and link it here once it’s updated.

Author: chriseth
Date: 2020-03-24T20:59:38.951Z

Can we just schedule a call tomorrow or on Thursday and settle most of the questions in on hour?

Author: njgheorghita
Date: 2020-03-24T21:29:08.964Z

Good call - let’s aim for 9AM CST / 3PM CET on Thursday. It’s flexible so if anybody would like to join but has a conflict - we can adjust the time as needed. I’ll post a zoom link here a couple minutes before the time we settle on.

Author: njgheorghita
Date: 2020-03-26T13:54:46.857Z

Here’s the zoom link for the call taking place in about 5 minutes
https://us04web.zoom.us/j/902568522?pwd=VlFtTldGNzFFcFpOdUVjOWdlcTFkdz09

Author: njgheorghita
Date: 2020-03-26T14:25:25.476Z

Updated zoom link
https://us04web.zoom.us/j/355814183?pwd=OW5uWWIvZjFUZDl6c3pOUkFZN0NYQT09

Author: chriseth
Date: 2020-03-26T15:54:32.075Z

Thanks for the call, that was very insightful!
During the call, ben was briefly talking about importing interfaces through their JSON-ABI as an interoperability measure. The relevant issue in the Solidity repo is here: https://github.com/ethereum/solidity/issues/1687
A comment by alex suggests to put these abi-json interfaces in their own files in the standard-io-input (as part of the “sources” field) and  add a “kind” field for each source file where this would be something like “abi-json”.
This “kind” field would also make it easier for projects spanning multiple languages.

Author: rumkin
Date: 2020-03-26T19:28:12.713Z

For interfaces import I’d suggest to have an ability of file import, like so:
interface Token from './erc20.json'

And for JSON files with custom structure it’s possible to define a path inside of JSON:
interface Token from './erc20.json#contracts[0].abi'

Where ./erc.json is a path to ABI file and #contracts[0].abi is a path inside of JSON object. It uses some trick as URL suppose hash wouldn’t be sent with a request and should be handled on the client, it’s safe to use hash in paths like so. And handle it with solidity compiler itself.

Author: njgheorghita
Date: 2020-04-07T00:14:17.220Z

@rumkin I’m not hugely familiar with how the solidity compiler handles imports - but this seems a little out of scope for the ethpm v3 specification. I could be wrong though, so feel free to bring this up at the next sync.

Author: njgheorghita
Date: 2020-04-07T00:23:09.556Z

Hey Guys! Hope everyone’s well.
The ethpm docs now reflect the changes we’ve discussed. http://ethpm.github.io/ethpm-spec/
There is now a machine-readable schema to validate v3 manifests - it can be found under spec/v3.spec.json in the ethpm-spec repo.
The examples in the ethpm-spec repo have been updated to reflect the changes in the spec - for a visual idea of what v3 packages might look like. You can find them under their respective folders under /examples/ with the file name v3.json @ https://github.com/ethpm/ethpm-spec
This spec is still a WIP - though we’ll be trying to finalize it within the next sync or two - so the sooner we get any suggested changes, the better. To give everyone enough time to go over the changes / examples, let’s plan our next sync for Thursday, April 16 - at 9AM CST / 3PM CET. If that time doesn’t conflict with anybody, I’ll post a zoom link here shortly before. Cheers!

Author: njgheorghita
Date: 2020-04-16T13:56:07.544Z

Here’s the link for the sync in 5 minutes.
https://us04web.zoom.us/j/79099681560?pwd=S01qaDFmb0c1Z1QvM1BvMTZXNVUxZz09

Author: njgheorghita
Date: 2020-04-30T00:45:00.443Z

Just a heads up, if you’re not already attending the solidity online conference. There’s going to be an ethpm discussion tomorrow at 8:55 pm central european time. The conversation will revolve around ethpm devx and the v3 technical specification. If you can make it, it’d be great to see everyone there!
https://interspace.solidity-summit.ethereum.org/

Author: njgheorghita
Date: 2020-04-30T19:35:07.840Z

Hey all - to continue with the discussion from today’s solidity summit we’ll have a call next Thursday, May 7 at 9AM CST / 3PM CET. I’ll post a zoom link here shortly before.

Author: njgheorghita
Date: 2020-05-06T20:39:34.257Z

General friendly reminder / ping about the sync tomorrow. It should be the last sync before finalizing the v3 spec! Hope to see y’all there - i’ll post the zoom link here shortly before start time.

Author: njgheorghita
Date: 2020-05-06T20:56:44.864Z

Sorry for any confusion, my timezone math was off. Correct times for the sync: 8AM CDT / 9AM EDT / 3PM CEST

Author: njgheorghita
Date: 2020-05-07T12:58:28.518Z

sync link: https://us04web.zoom.us/j/73933580804?pwd=Q3MrSDUwd2IzUVZLSUs2VDJuMnpkdz09

Author: jpitts
Date: 2020-05-07T21:26:10.370Z

Would you consider this a “last call” at this point? I will Tweet something out to see if there are others who would like to submit feedback.

Author: fubuloubu
Date: 2020-05-08T14:17:31.612Z

Yeah, @njgheorghita would it be possible to draft a blog post about this so we can Tweet it out and get Last Call comments on it before you draft the EIP?

Author: njgheorghita
Date: 2020-05-08T14:22:14.817Z

Yeah, that would be excellent. Tweet blasts are much appreciated as my twitter game is pretty weak. I’ll type it up and link it here as soon as it’s ready.

Author: njgheorghita
Date: 2020-05-11T21:31:12.857Z

ping @fubuloubu @jpitts. I wrote up a short outline of the proposed v3 changes here - thanks for the blast!
https://medium.com/@nickg_82582/ethpm-v3-is-coming-500d0edd38c4?sk=79441c8f492b9d43186063915ffc7594
Let me know if I forgot something or got anything horribly wrong.
I’m also proposing a spec freeze date of May 18 - which can be delayed if there are any problems found.

Author: MicahZoltu
Date: 2020-12-17T18:22:07.903Z

Moving conversation here @njgheorghita since this isn’t really about the EIP structure (which is all that should really be discussed in PRs).


I’m specifically referring to the source checksum which explicitly says that it is required if there is no URI (meaning the file is not available on a content-addressable filesystem). Either you have a URI like ipfs://… or swarm://…, or you don’t. In the case that you do, the checksum field is redundant. In the case that you don’t, the checksum field is useless.

In the case that you don’t have a URI - it is required to include the contract source as a string (under the “content” key) in which case the checksum field is also required, containing a checksum object with the algorithm used to generate a hash and the resolved hash of the inlined source.

What is the purpose of including the checksum in this scenario?  What problem does it solve?  What value does it provide to the user?

Author: njgheorghita
Date: 2020-12-21T15:50:46.495Z

If my memory serves correctly, this requirement was enforced to enable tooling / frameworks to perform additional verification of any in-lined source assets. It is possible for ethpm packages to be imported solely as a json file (for example… if a user copy/pastes the json directly from an ipfs explorer rather than importing via the ipfs uri), in which case requiring the checksum provides some extra confidence in the authenticity of the inlined sources. However, if a user is interacting with the ethpm ecosystem as it was intended, then this requirement is overkill. I’d be on board to update the requirement to include a Checksum object along with inlined sources to just a recommendation, unless there are other objections.

Author: MicahZoltu
Date: 2020-12-22T04:08:47.916Z

Why not checksum the entire file in that case, as is common practice throughout the industry (serving a file hash as well as the file itself to protect users from MITM attacks)?  This way, users in such a scenario can use existing tooling rather than needing custom tooling.  This also would mean the entire file would be checksummed, rather than just part of the file.

Author: fubuloubu
Date: 2021-01-08T18:15:23.917Z

Just mandate the IPFS CID, and then allow an alternative link that would give the same file as if you downloaded it from IPFS

Author: njgheorghita
Date: 2021-01-09T18:26:04.716Z

Just to have some more concrete examples to work with here, these are the three ways that the spec currently allows users to include a source.

In-lined source requires a Checksum object

{"sources": {
	"Owned.sol": {
		"content": "pragma ...",
		"checksum": {
			"algorithm": "sha256",
			"hash": "a1b2c3..."
		}
	},
}}


Source identified by a content-addressed URI does not require Checksum object

{"sources": {
	"Owned.sol": {
		"uls": ["ipfs://Qm..."]
	}
}}


Source identified by a generic URI (aka a URI without a checksum of its contents) requires a Checksum object.

{"sources": {
	"Owned.sol": {
		"uls": ["github.com/myproject/Owned.sol"],
		"checksum": {
			"algorithm": "sha256",
			"hash": "a1b2c3..."
		}
	}
}}

@fubuloubu If I understand your example correctly, it would fall under the third category where the package creator could populate the Checksum object with the ipfs cid.
@MicahZoltu Correct me if I’m wrong, your concern is that in #1 requiring the Checksum object is redundant. I’m ok with dropping the requirement for a Checksum object to a recommendation in the first scenario.

Author: MicahZoltu
Date: 2021-01-10T07:08:24.452Z

njgheorghita:

Correct me if I’m wrong, your concern is that in #1 requiring the Checksum object is redundant. I’m ok with dropping the requirement for a Checksum object to a recommendation in the first scenario.


Yes, in (1) the checksum provides no value because the payload source and the checksum source are the same which means it is trivial to put in a checksum that matches the provided payload.
In (2) no checksum is needed.
In (3) a checksum is valuable and I am OK with it being required there.

Author: njgheorghita
Date: 2021-02-03T22:30:27.154Z

Sorry for the delay - I’ve updated the EIP here.

Author: fubuloubu
Date: 2021-02-11T22:55:08.988Z

Is this ready for Last Call?

Author: njgheorghita
Date: 2021-02-16T12:09:23.335Z

Last Call pr is up! https://github.com/ethereum/EIPs/pull/3269
Review period ends March 2. If there are any last minute suggestions / amendments to discuss please post them here asap!

Author: ShadeUndertree
Date: 2021-03-10T18:30:12.015Z

I implemented a basic version of this spec for usage with brownie / vyper compilation, and overall I found it fairly natural and straightforward, the team clearly put a lot of thought into developing this spec.
The only thing that was a bit confusing for me was the installPath property of the Source object. This isn’t needed for Vyper and I think this may be Solidity specific? It could be helpful to add a comment to the spec specifying anything that’s specific to one language’s compiler vs. another, etc.
Alternatively, another option would be to adjust the name of installPath to something like ‘LocalPath’ or ‘CachePath’. Using Vyper I just compile from an in-memory string, whereas with Solidity this doesn’t seem possible hence the need for this property, if I’m understanding correctly.

Author: esaulpaugh
Date: 2021-03-11T02:23:53.813Z

I never understood the point of the 0x prefix. It seems like an attempt to mix type information into the data itself. Ironically, something like “0x44” is valid Base64 and valid ASCII and a million other things, yet ‘x’ is not a valid hex character. So decoders have to decide whether to reject it or require it, or allow it optionally but check whether it’s actually there). I don’t think it improves readability by computers or humans. And certainly not interoperability.
‘0x’ is good for programming languages where multiple represenations can be used when assigning a variable. Multiple representations for the same field seems like a bad idea inside a standard.

Author: njgheorghita
Date: 2021-04-08T14:46:59.031Z

Agreed that installPath is not the most intuitive design. If I’m remembering correctly, we went with this design in order to preserve the rule that the key in the sources object can be any unique string - rather than the filepath itself. You’re correct, this is important for compatibility with the solidity compiler output, which was one of the primary design goals for v3. Thanks for bringing this up though, it’s really helpful feedback. I’ll add a note to the spec, but personally, I’m not convinced that it’s worth changing the spec. If you or somebody else feels strongly about this, I’m happy to give it further consideration.

Author: njgheorghita
Date: 2021-04-08T14:54:49.921Z

The 0x prefix is used to identify that the following string is hex encoded. It’s fairly standard encoding practice across the ethereum ecosystem. The spec and tooling supports unprefixed hex-strings, but specifies that all hex-strings should use the 0x prefix for consistency. Personally, I always use 0x prefixed hex-strings and rarely see unprefixed hex-strings in the wild. The spec also specifies that the package must use utf-8 encoding. Tooling shouldn’t have a problem decoding hex-strings according to the spec, all that’s needed is a simple check whether or not hex-string fields have the 0x prefix, and then handle those fields accordingly. I’m not eager to change the hex-string encoding conventions of the spec at this point, especially since it’s in-line with broad standard practice in the community and I’ve never heard concerns from tooling developers or users.

Author: ShadeUndertree
Date: 2021-04-08T15:17:46.187Z

I’d be quite happy with a note added to the spec. I understand not changing the spec itself at this point in the process. Thank you very much for your consideration on my feedback!

Author: esaulpaugh
Date: 2021-04-11T22:42:49.555Z

Bouncycastle and apache commons (commons-codec) both barf on 0x (rightly, in my opinion). Seems strange that ethereum would require a bespoke implementation of hexadecimal decoding.
I do know that cryptographic hashes are by convention never prefixed with 0x. So keccak256 fields are violating that convention.