Repository: goplus/gop
Pull request #506 (2020-07-28 14:56:03)

cmd/qexp: support auto download modules. support pkg/... for export all in pkg dir.
export support check and auto download modules. (use go mod download)
but unsupported replace module and skip.

support pkg/...
```
>qexp unicode/...
source c:\go\src\unicode
export "unicode" success
export "unicode/utf16" success
export "unicode/utf8" success
```
support auto download mod
```
>qexp github.com/visualfc/fastmod@latest
exec: go download mod github.com/visualfc/fastmod@latest
source C:\Users\HOME\go\pkg\mod\github.com\visualfc\fastmod@v1.3.6
export "github.com/visualfc/fastmod" success
```
no version lookup for local gomod
```
>qexp github.com/visualfc/fastmod
source C:\Users\HOME\go\pkg\mod\github.com\visualfc\fastmod@v1.3.6
export "github.com/visualfc/fastmod" success
```
audo download mod with sub package
```
>qexp github.com/visualfc/gotools@master/pkg/godiff
exec: go download mod github.com/visualfc/gotools@master
source C:\Users\HOME\go\pkg\mod\github.com\visualfc\gotools@v1.3.3
export "github.com/visualfc/gotools/pkg/godiff" success
```
auto download mod with sub package/...
```
>qexp github.com/visualfc/gotools@master/pkg/...
exec: go download mod github.com/visualfc/gotools@master
source C:\Users\HOME\go\pkg\mod\github.com\visualfc\gotools@v1.3.3
export "github.com/visualfc/gotools/pkg/buildctx" success
export "github.com/visualfc/gotools/pkg/command" success
export "github.com/visualfc/gotools/pkg/godiff" success
export "github.com/visualfc/gotools/pkg/gomod" success
export "github.com/visualfc/gotools/pkg/pkgutil" success
export "github.com/visualfc/gotools/pkg/pkgwalk" success
export "github.com/visualfc/gotools/pkg/srcimporter" success
export "github.com/visualfc/gotools/pkg/stdlib" success
```
_current unsupport replace module, check and skip._
```
>qexp.exe github.com/visualfc/gopherjs@latest/build
exec: go download mod github.com/visualfc/gopherjs@master
lookup "github.com/visualfc/gopherjs@master" source error: unsupport replace module "github.com/gopherjs/gopherjs"
```





codecov-commenter: # [Codecov](https://codecov.io/gh/goplus/gop/pull/506?src=pr&el=h1) Report
> Merging [#506](https://codecov.io/gh/goplus/gop/pull/506?src=pr&el=desc) into [master](https://codecov.io/gh/goplus/gop/commit/9a596ca22463f116cb78f9876b2c2a371d225066&el=desc) will **increase** coverage by `0.20%`.
> The diff coverage is `58.33%`.

[![Impacted file tree graph](https://codecov.io/gh/goplus/gop/pull/506/graphs/tree.svg?width=650&height=150&src=pr&token=4qln9Wk3Pn)](https://codecov.io/gh/goplus/gop/pull/506?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #506      +/-   ##
==========================================
+ Coverage   89.07%   89.27%   +0.20%     
==========================================
  Files          39       40       +1     
  Lines        9791     9784       -7     
==========================================
+ Hits         8721     8735      +14     
+ Misses        787      765      -22     
- Partials      283      284       +1     
```


| [Impacted Files](https://codecov.io/gh/goplus/gop/pull/506?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [cmd/internal/export/gopkg/export.go](https://codecov.io/gh/goplus/gop/pull/506/diff?src=pr&el=tree#diff-Y21kL2ludGVybmFsL2V4cG9ydC9nb3BrZy9leHBvcnQuZ28=) | `86.27% <58.33%> (-2.24%)` | :arrow_down: |
| [exec/bytecode/goinstr.go](https://codecov.io/gh/goplus/gop/pull/506/diff?src=pr&el=tree#diff-ZXhlYy9ieXRlY29kZS9nb2luc3RyLmdv) | `92.70% <0.00%> (-1.02%)` | :arrow_down: |
| [exec/bytecode/const.go](https://codecov.io/gh/goplus/gop/pull/506/diff?src=pr&el=tree#diff-ZXhlYy9ieXRlY29kZS9jb25zdC5nbw==) | `90.10% <0.00%> (-0.22%)` | :arrow_down: |
| [exec/golang/builder.go](https://codecov.io/gh/goplus/gop/pull/506/diff?src=pr&el=tree#diff-ZXhlYy9nb2xhbmcvYnVpbGRlci5nbw==) | `91.08% <0.00%> (-0.12%)` | :arrow_down: |
| [cl/expr.go](https://codecov.io/gh/goplus/gop/pull/506/diff?src=pr&el=tree#diff-Y2wvZXhwci5nbw==) | `84.04% <0.00%> (-0.08%)` | :arrow_down: |
| [exec/bytecode/code.go](https://codecov.io/gh/goplus/gop/pull/506/diff?src=pr&el=tree#diff-ZXhlYy9ieXRlY29kZS9jb2RlLmdv) | `100.00% <0.00%> (ø)` | |
| [exec/bytecode/defer.go](https://codecov.io/gh/goplus/gop/pull/506/diff?src=pr&el=tree#diff-ZXhlYy9ieXRlY29kZS9kZWZlci5nbw==) | `100.00% <0.00%> (ø)` | |
| [exec/bytecode/stack.go](https://codecov.io/gh/goplus/gop/pull/506/diff?src=pr&el=tree#diff-ZXhlYy9ieXRlY29kZS9zdGFjay5nbw==) | `100.00% <0.00%> (ø)` | |
| [exec/bytecode/interface.go](https://codecov.io/gh/goplus/gop/pull/506/diff?src=pr&el=tree#diff-ZXhlYy9ieXRlY29kZS9pbnRlcmZhY2UuZ28=) | `99.04% <0.00%> (+<0.01%)` | :arrow_up: |
| [exec/golang/stmt.go](https://codecov.io/gh/goplus/gop/pull/506/diff?src=pr&el=tree#diff-ZXhlYy9nb2xhbmcvc3RtdC5nbw==) | `95.71% <0.00%> (+0.01%)` | :arrow_up: |
| ... and [7 more](https://codecov.io/gh/goplus/gop/pull/506/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/goplus/gop/pull/506?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/goplus/gop/pull/506?src=pr&el=footer). Last update [ff872db...c3e78ac](https://codecov.io/gh/goplus/gop/pull/506?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/cmd/internal/export/export.go b/cmd/internal/export/export.go
index 28725819..11d46084 100644
--- a/cmd/internal/export/export.go
+++ b/cmd/internal/export/export.go
@@ -19,11 +19,15 @@ package export
 
 import (
 	"bytes"
+	"encoding/json"
 	"fmt"
 	"go/format"
+	"go/types"
 	"io/ioutil"
 	"os"
+	"os/exec"
 	"path/filepath"
+	"strings"
 
 	"github.com/goplus/gop/cmd/internal/base"
 	"github.com/goplus/gop/cmd/internal/export/gopkg"
@@ -50,13 +54,25 @@ func init() {
 	Cmd.Run = runCmd
 }
 
+var (
+	libDir string
+	gobin  string
+)
+
+func init() {
+	var err error
+	gobin, err = exec.LookPath("go")
+	if err != nil {
+		panic("not found go bin in PATH")
+	}
+}
+
 func runCmd(cmd *base.Command, args []string) {
 	flag.Parse(args)
 	if flag.NArg() < 1 {
 		cmd.Usage(os.Stderr)
 		return
 	}
-	var libDir string
 	if flagExportDir != "" {
 		libDir = flagExportDir
 	} else {
@@ -69,20 +85,162 @@ func runCmd(cmd *base.Command, args []string) {
 	}
 
 	for _, pkgPath := range flag.Args() {
-		err := exportPkg(pkgPath, libDir)
+		var exporAll bool
+		if strings.HasSuffix(pkgPath, "/...") {
+			pkgPath = pkgPath[:len(pkgPath)-4]
+			exporAll = true
+		}
+		pkg, err := LookupPkg(pkgPath)
+		if err != nil {
+			fmt.Printf("lookup %q source error: %v\n", pkgPath, err)
+			continue
+		}
+		fmt.Printf("source %v\n", pkg.Dir)
+		if exporAll {
+			exportDir(pkg.ImportPath, pkg.Dir)
+			continue
+		}
+		if pkg.Goroot {
+			err = exportPkg(pkg.ImportPath, "")
+		} else {
+			err = exportPkg(pkg.ImportPath, pkg.Dir)
+		}
 		if err != nil {
-			fmt.Fprintf(os.Stderr, "export pkg %q failed, %v\n", pkgPath, err)
+			fmt.Printf("export %q error: %v\n", pkg.ImportPath, err)
 		} else {
-			fmt.Fprintf(os.Stdout, "export pkg %q success\n", pkgPath)
+			fmt.Printf("export %q success\n", pkg.ImportPath)
+		}
+	}
+}
+
+type Package struct {
+	Dir        string // directory containing package sources
+	ImportPath string // import path of package in dir
+	Goroot     bool   // is this package in the Go root?
+}
+
+func LookupPkg(pkgPath string) (*Package, error) {
+	// check go list
+	if !strings.Contains(pkgPath, "@") {
+		pkg, err := checkGoList(pkgPath)
+		if err == nil {
+			return pkg, nil
 		}
 	}
+	// check go mod
+	srcDir, err := gopkg.LookupMod(pkgPath)
+	if err == nil {
+		pkgPath, _, _ = gopkg.ParsePkgVer(pkgPath)
+		rpkg, _ := checkGoPkg(srcDir)
+		if rpkg != pkgPath {
+			return nil, fmt.Errorf("unsupport replace module %q", rpkg)
+		}
+		return &Package{ImportPath: pkgPath, Dir: srcDir}, nil
+	}
+	// check download mod
+	pkg, mod, sub := gopkg.ParsePkgVer(pkgPath)
+	info, err := downloadMod(mod)
+	if err != nil {
+		return nil, fmt.Errorf("download %q failed, %v", mod, err)
+	}
+	dir := filepath.Join(info.Dir, sub)
+	rpkg, _ := checkGoPkg(info.Dir)
+	if rpkg != info.Path {
+		return nil, fmt.Errorf("unsupport replace module %q", rpkg)
+	}
+	return &Package{ImportPath: pkg, Dir: dir}, nil
+}
+
+type ModInfo struct {
+	Path     string // module path
+	Version  string // module version
+	Error    string // error loading module
+	Info     string // absolute path to cached .info file
+	GoMod    string // absolute path to cached .mod file
+	Zip      string // absolute path to cached .zip file
+	Dir      string // absolute path to cached source root directory
+	Sum      string // checksum for path, version (as in go.sum)
+	GoModSum string // checksum for go.mod (as in go.sum)
+}
+
+func downloadMod(pkgPath string) (*ModInfo, error) {
+	fmt.Println("exec: go download mod", pkgPath)
+	cmd := exec.Command(gobin, "mod", "download", "-json", pkgPath)
+	data, err := cmd.Output()
+	if err != nil {
+		return nil, err
+	}
+	var info ModInfo
+	err = json.Unmarshal(data, &info)
+	if err != nil {
+		return nil, err
+	}
+	return &info, nil
 }
 
-func exportPkg(pkgPath string, libDir string) error {
-	pkg, err := gopkg.Import(pkgPath)
+func exportDir(pkgPath string, dir string) error {
+	cmd := exec.Command(gobin, "list", "-f", "{{.ImportPath}} {{.Dir}}", "./...")
+	cmd.Dir = dir
+	data, err := cmd.Output()
 	if err != nil {
 		return err
 	}
+	lines := strings.Split(string(data), "\n")
+	for _, line := range lines {
+		ar := strings.Split(line, " ")
+		if len(ar) == 2 {
+			pkg, dir := ar[0], ar[1]
+			err := exportPkg(pkg, dir)
+			if strings.Contains(pkg, "internal") || strings.Contains(pkg, "vendor") {
+				continue
+			}
+			if err == nil {
+				fmt.Printf("export %q success\n", pkg)
+			} else if err != gopkg.ErrIgnore {
+				fmt.Printf("export %v error,%v\n", pkg, err)
+			}
+		}
+	}
+	return nil
+}
+
+func checkGoList(pkgPath string) (*Package, error) {
+	cmd := exec.Command(gobin, "list", "-json", pkgPath)
+	data, err := cmd.Output()
+	if err != nil {
+		return nil, err
+	}
+	var pkg Package
+	err = json.Unmarshal(data, &pkg)
+	if err != nil {
+		return nil, err
+	}
+	return &pkg, nil
+}
+
+func checkGoPkg(srcDir string) (string, error) {
+	cmd := exec.Command(gobin, "list")
+	cmd.Dir = srcDir
+	data, err := cmd.Output()
+	if err != nil {
+		return "", err
+	}
+	return strings.TrimSpace(string(data)), nil
+}
+
+func exportPkg(pkgPath string, srcDir string) (err error) {
+	var pkg *types.Package
+	if srcDir == "" {
+		pkg, err = gopkg.Import(pkgPath)
+	} else {
+		pkg, err = gopkg.ImportSource(pkgPath, srcDir)
+	}
+	if err != nil {
+		return fmt.Errorf("import %q failed: %v", pkgPath, err)
+	}
+	if pkg.Name() == "main" {
+		return gopkg.ErrIgnore
+	}
 	var buf bytes.Buffer
 	err = gopkg.ExportPackage(pkg, &buf)
 	if err != nil {
@@ -96,7 +254,7 @@ func exportPkg(pkgPath string, libDir string) error {
 	dir := filepath.Join(libDir, pkg.Path())
 	os.MkdirAll(dir, 0777)
 	err = ioutil.WriteFile(filepath.Join(dir, "gomod_export.go"), data, 0666)
-	return err
+	return
 }
 
 // -----------------------------------------------------------------------------
diff --git a/cmd/internal/export/gopkg/export.go b/cmd/internal/export/gopkg/export.go
index 8228af7f..571cf662 100644
--- a/cmd/internal/export/gopkg/export.go
+++ b/cmd/internal/export/gopkg/export.go
@@ -64,16 +64,18 @@ func exportConst(e *Exporter, o *types.Const) (err error) {
 	return e.ExportConst(o)
 }
 
-func cleanPkgVer(pkgPath string) string {
+// ParsePkgVer
+// github.com/qiniu/x@v1.11.5/log => github.com/qiniu/x/@v1.11.5 github.com/qiniu/x/log log
+func ParsePkgVer(pkgPath string) (pkg string, mod string, sub string) {
 	i := strings.Index(pkgPath, "@")
 	if i == -1 {
-		return pkgPath
+		return pkgPath, "", ""
 	}
 	j := strings.Index(pkgPath[i:], "/")
 	if j == -1 {
-		return pkgPath[:i]
+		return pkgPath[:i], pkgPath, ""
 	}
-	return pkgPath[:i] + pkgPath[i:][j:]
+	return pkgPath[:i] + pkgPath[i:][j:], pkgPath[:i+j], pkgPath[i+j+1:]
 }
 
 func findLastVerPkg(pkgDirBase string, name string) (verName string) {
@@ -138,9 +140,16 @@ func LookupMod(pkgPath string) (srcDir string, err error) {
 	}
 	srcDir = filepath.Join(srcDir, parts[2])
 	if n > 3 {
-		srcDir += "/" + strings.Join(parts[3:], "/")
+		srcDir = filepath.Join(srcDir, filepath.Join(parts[3:]...))
 	}
-	return srcDir, nil
+	s, err := os.Stat(srcDir)
+	if err != nil {
+		return "", err
+	}
+	if !s.IsDir() {
+		return "", os.ErrInvalid
+	}
+	return
 }
 
 // ImportSource import a Go package from pkgPath and srcDir
@@ -165,7 +174,7 @@ func Import(pkgPath string) (*types.Package, error) {
 		if noVer {
 			parts[2] = findLastVerPkg(srcDir, parts[2])
 		} else {
-			pkgPath = cleanPkgVer(pkgPath)
+			pkgPath, _, _ = ParsePkgVer(pkgPath)
 		}
 		srcDir = filepath.Join(srcDir, parts[2])
 		if n > 3 {


Test patch:
diff --git a/cmd/internal/export/gopkg/exporter_test.go b/cmd/internal/export/gopkg/exporter_test.go
index 7f23caa5..d54280ed 100644
--- a/cmd/internal/export/gopkg/exporter_test.go
+++ b/cmd/internal/export/gopkg/exporter_test.go
@@ -193,6 +193,22 @@ func TestLooupMod(t *testing.T) {
 	if dir != dir1 {
 		t.Fatal("LookupMod failed:", dir, dir1)
 	}
+
+}
+
+func TestBadLooupMod(t *testing.T) {
+	dir, err := LookupMod("github.com/qiniu/x/log2")
+	if err == nil {
+		t.Fatal("LookupMod bad:", dir)
+	}
+	dir, err = LookupMod("github.com/qiniu/x/doc.go")
+	if err == nil {
+		t.Fatal("LookupMod bad:", dir)
+	}
+	dir, err = LookupMod("unicode")
+	if err == nil {
+		t.Fatal("LookupMod bad:", dir)
+	}
 }
 
 func TestExportVersion(t *testing.T) {
@@ -208,14 +224,36 @@ func TestExportVersion(t *testing.T) {
 	}
 }
 
-func TestCleanVersion(t *testing.T) {
-	pkg := cleanPkgVer("github.com/qiniu/x@v1.11.5/log")
+func TestParsePkgVer(t *testing.T) {
+	pkg, mod, sub := ParsePkgVer("github.com/qiniu/x@v1.11.5/log")
 	if pkg != "github.com/qiniu/x/log" {
-		t.Fatal("cleanPkgVer failed:", pkg)
+		t.Fatal("ParsePkgVer failed:", pkg)
 	}
-	pkg = cleanPkgVer("github.com/qiniu/x@v1.11.5")
+	if mod != "github.com/qiniu/x@v1.11.5" {
+		t.Fatal("ParsePkgVer failed:", mod)
+	}
+	if sub != "log" {
+		t.Fatal("ParsePkgVer failed:", sub)
+	}
+	pkg, mod, sub = ParsePkgVer("github.com/qiniu/x@v1.11.5")
+	if pkg != "github.com/qiniu/x" {
+		t.Fatal("ParsePkgVer failed:", pkg)
+	}
+	if mod != "github.com/qiniu/x@v1.11.5" {
+		t.Fatal("ParsePkgVer failed:", mod)
+	}
+	if sub != "" {
+		t.Fatal("ParsePkgVer failed:", sub)
+	}
+	pkg, mod, sub = ParsePkgVer("github.com/qiniu/x")
 	if pkg != "github.com/qiniu/x" {
-		t.Fatal("cleanPkgVer failed:", pkg)
+		t.Fatal("ParsePkgVer failed:", pkg)
+	}
+	if mod != "" {
+		t.Fatal("ParsePkgVer failed:", mod)
+	}
+	if sub != "" {
+		t.Fatal("ParsePkgVer failed:", sub)
 	}
 }