Repository: cloud-custodian/tfparse (language: python)

Issue:
Document local dev and add some basic scripts for working locally (makefile/justfile/whatever)
It'd be nice to have some docs on:

1. How to build the go/python to make sure its working
2. How to setup your vscode so that it can debug the go 
3. How to run tests and all that

If we have a makefile that does this it'd be great, but docs are good enough.


Discussion / hints:
We don't use just anywhere else in cloud-custodian projects. I don't know how I feel about forcing folks to need yet another tool .. probably just getting some niceties setup in setup.py / pyproject.toml is sufficient.
Makefile is more standard in cloud-custodian, albeit I ❤️ `just` but its hard to justify 😉  since its not a standard tool, its getting close though to easy, one notable gap towards that it still needs to make its way into Debian/ubuntu upstream package universe though.

Fix patch:
diff --git a/setup.cfg b/.flake8
similarity index 50%
rename from setup.cfg
rename to .flake8
index 3b14ff4..e44645b 100644
--- a/setup.cfg
+++ b/.flake8
@@ -1,4 +1,3 @@
 [flake8]
 max-line-length = 100
-exclude = tfparse/_tfparse.py
-
+exclude = "tfparse/_tfparse.py"
\ No newline at end of file
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index 01c05c2..fe7bbed 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -21,7 +21,7 @@ jobs:
         uses: pypa/cibuildwheel@7c45799919d2dcd7ac59433924d763dd24d97483 # ratchet:pypa/cibuildwheel@v2.9.0
         env:
           CIBW_ENVIRONMENT: PATH=$(pwd)/go/bin:$PATH
-          CIBW_BEFORE_ALL: sh setup-golang.sh
+          CIBW_BEFORE_ALL: sh ci-setup-golang.sh
           CIBW_SKIP: "*musllinux*"
           CIBW_ARCHS: "x86_64 aarch64"
       - uses: actions/upload-artifact@3cea5372237819ed00197afe530f5a7ea3e805c8 # ratchet:actions/upload-artifact@v3
@@ -43,7 +43,7 @@ jobs:
       - name: Set up Go
         uses: actions/setup-go@268d8c0ca0432bb2cf416faae41297df9d262d7f # ratchet:actions/setup-go@v3
         with:
-          go-version: "1.18"
+          go-version: "1.19"
           cache: true
           cache-dependency-path: "gotfparse/go.sum"
       - name: Build wheels
diff --git a/.gitignore b/.gitignore
index 0526a9b..65a08b7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,3 +8,10 @@ venv/**
 *.so
 .python-version
 __pycache__
+poetry.lock
+tfparse.cpython*
+.vscode
+
+.terraform
+.tf_providers
+.terraform.lock.hcl
\ No newline at end of file
diff --git a/README.md b/README.md
index fd02b05..c95d8e7 100644
--- a/README.md
+++ b/README.md
@@ -29,17 +29,45 @@ parsed = load_from_path('path_to_terraform_root')
 print(parsed.keys())
 ```
 
-# Building
+# Developing
 
-- requires a modern golang (1.18)
+- requires Go >= 1.18
+- requires Python >= 3.10
+
+This project uses [Poetry][poetry_website] for package management, if you do not already have Poetry installed you can do so by running the following command:
+
+    curl -sSL https://install.python-poetry.org | python3 -
+
+
+
+## Installing from source
+
+Installing will build the module and install the local copy of tfparse in to the current Python environment.
 
 ```shell
-python setup.py develop
+poetry install
+python
+>>> from tfparse import load_from_path
+>>> parsed = load_from_path('<path_to_terraform>')
+>>> print(parsed.keys())
 ```
 
+## Building from source
 
-## Credits
+Building will produce a wheel and a source artifact for distribution or upload to package repositories.
+
+```shell
+poetry build
+ls -l dist/
+```
+
+
+# Credits
 
 aquasecurity/defsec - golang module for parsing and evaluating terraform hcl
 
 Scalr/pygohcl - python bindings for terraform hcl via golang extension
+
+# Links
+
+[poetry_website]: https://python-poetry.org/
\ No newline at end of file
diff --git a/build.py b/build.py
new file mode 100644
index 0000000..3957fc5
--- /dev/null
+++ b/build.py
@@ -0,0 +1,43 @@
+#!/usr/bin/env python
+# Copyright The Cloud Custodian Authors.
+# SPDX-License-Identifier: Apache-2.0
+
+from distutils.command.build_ext import build_ext
+from distutils.errors import CCompilerError, DistutilsExecError, DistutilsPlatformError
+from distutils.extension import Extension
+
+ext_modules = [Extension("tfparse", ["gotfparse/cmd/tfparse/main.go"])]
+cffi_modules = ["tfparse/build_cffi.py:ffi"]
+build_golang = {"root": "github.com/cloud-custodian/tfparse/gotfparse"}
+
+
+class BuildFailed(Exception):
+    pass
+
+
+class ExtBuilder(build_ext):
+    def run(self):
+        try:
+            build_ext.run(self)
+        except (DistutilsPlatformError, FileNotFoundError):
+            raise BuildFailed("File not found. Could not compile C extension.")
+
+    def build_extension(self, ext):
+        try:
+            build_ext.build_extension(self, ext)
+        except (CCompilerError, DistutilsExecError, DistutilsPlatformError, ValueError):
+            raise BuildFailed("Could not compile C extension.")
+
+
+def build(setup_kwargs):
+    """
+    This function is mandatory in order to build the extensions.
+    """
+    setup_kwargs.update(
+        {
+            "build_golang": build_golang,
+            "ext_modules": ext_modules,
+            "cffi_modules": cffi_modules,
+            "cmdclass": {"build_ext": ExtBuilder},
+        }
+    )
diff --git a/setup-golang.sh b/ci-setup-golang.sh
similarity index 78%
rename from setup-golang.sh
rename to ci-setup-golang.sh
index faeb873..3157f99 100755
--- a/setup-golang.sh
+++ b/ci-setup-golang.sh
@@ -2,7 +2,7 @@
 
 OS=$(uname -s)
 ARCH=$(uname -m)
-GOVER="1.18.5"
+GOVER="1.19.2"
 case $ARCH in
     x86_64) ARCH="amd64" ;;
     aarch64) ARCH="arm64" ;;
@@ -12,7 +12,7 @@ case $OS in
     Linux) OS="linux" ;;
 esac    
 	    
-curl "https://storage.googleapis.com/golang/go1.18.5.${OS}-${ARCH}.tar.gz" --silent --location | tar -xz
+curl "https://storage.googleapis.com/golang/go1.19.2.${OS}-${ARCH}.tar.gz" --silent --location | tar -xz
 
 export PATH="$(pwd)/go/bin:$PATH"
 
diff --git a/gotfparse/README.md b/gotfparse/README.md
index e25013c..2742ebf 100644
--- a/gotfparse/README.md
+++ b/gotfparse/README.md
@@ -1,3 +1,19 @@
 # gotfparse
 
-`gotfparse` is a Go library that wraps the super useful <link-to-defsec> parser. This is done to provide an exported function to Python that takes advantage of the capabilities and speed of the defsec HCL parser.
\ No newline at end of file
+`gotfparse` is a Go library that wraps the [defsec][defsec_repo] parser. This is done to provide an exported function to Python that takes advantage of the capabilities and speed of the defsec HCL parser.
+
+# Developing
+
+    go mod tidy
+
+You can use the `tftest` helper command to easily iterate on terraform and preview the JSON output that the `gotfparse` library produces.
+
+    go run cmd/tftest/main.go <path-to-terraform> > output.json
+
+## Tips
+
+When using a modern IDE like Visual Studio Code or Goland, open the `gotfparse` folder as the root of the workspace to ensure all of the Go tooling works as expected.
+
+# Links
+
+[defsec_repo]: https://github.com/aquasecurity/defsec
\ No newline at end of file
diff --git a/pyproject.toml b/pyproject.toml
index b58ff70..95659ad 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,3 +1,40 @@
+
+[tool.poetry]
+name = "tfparse"
+version = "0.1.4"
+description = "Python HCL/Terraform parser via extension for AquaSecurity defsec"
+authors = ["Wayne Witzel III <wayne@stacklet.io>"]
+maintainers = ["Cloud Custodian Project <cloud-custodian@googlegroups.com>"]
+repository = "https://github.com/cloud-custodian/tfparse"
+license = "Apache-2.0"
+readme = "README.md"
+include = [
+    "LICENSE",
+]
+classifiers=[
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3.10",
+    "License :: OSI Approved :: Apache Software License",
+]
+build = "build.py"
+
+[tool.poetry.dependencies]
+python = "^3.8"
+cffi = ">=1.0.0"
+
+[tool.poetry.dev-dependencies]
+cffi = "^1.15.1"
+black = "^22.10.0"
+pytest = "^7.1.3"
+pytest-forked = "^1.4.0"
+pytest-terraform = "^0.6.4"
+pytest-xdist = "^2.5.0"
+flake8 = "^5.0.4"
+setuptools-golang = "^2.7.0"
+wheel = "^0.37.1"
+setuptools = "^65.4.1"
+
 [build-system]
-requires = ["setuptools", "wheel", "cffi >= 1.15.0"]
-build-backend = "setuptools.build_meta"
+requires = ["poetry-core>=1.0.0", "wheel", "setuptools", "setuptools-golang", "cffi >= 1.15.0"]
+build-backend = "poetry.core.masonry.api"
\ No newline at end of file
diff --git a/setup.py b/setup.py
deleted file mode 100644
index 8562f5d..0000000
--- a/setup.py
+++ /dev/null
@@ -1,38 +0,0 @@
-#!/usr/bin/env python
-# Copyright The Cloud Custodian Authors.
-# SPDX-License-Identifier: Apache-2.0
-
-from setuptools import Extension, find_packages, setup
-from pathlib import Path
-
-
-README = (Path(__file__).parent / "README.md").read_text()
-
-
-setup(
-    name="tfparse",
-    description="Python HCL/Terraform parser via extension for AquaSecurity defsec",
-    long_description=README,
-    long_description_content_type="text/markdown",
-    url="https://github.com/cloud-custodian/tfparse",
-    author="Wayne Witzel III",
-    author_email="wayne@stacklet.io",
-    maintainer="Cloud Custodian Project",
-    maintainer_email="cloud-custodian@googlegroups.com",
-    license="Apache-2.0",
-    version="0.1.4",
-    classifiers=[
-        "Development Status :: 3 - Alpha",
-        "Intended Audience :: Developers",
-        "Programming Language :: Python :: 3.10",
-        "License :: OSI Approved :: Apache Software License",
-    ],
-    packages=find_packages(),
-    install_requires=["cffi>=1.0.0"],
-    setup_requires=["cffi>=1.0.0", "setuptools-golang"],
-    build_golang={"root": "github.com/cloud-custodian/tfparse/gotfparse"},
-    ext_modules=[Extension("tfparse", ["gotfparse/cmd/tfparse/main.go"])],
-    cffi_modules=[
-        "tfparse/build_cffi.py:ffi",
-    ],
-)


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 0f5ef69..fae257e 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -42,7 +42,7 @@ jobs:
       - name: Set up Go
         uses: actions/setup-go@268d8c0ca0432bb2cf416faae41297df9d262d7f # ratchet:actions/setup-go@v3
         with:
-          go-version: "1.18"
+          go-version: "1.19"
           cache: true
           cache-dependency-path: "gotfparse/go.sum"
       - name: Install dependencies
diff --git a/gotfparse/cmd/tftest/README.md b/gotfparse/cmd/tftest/README.md
new file mode 100644
index 0000000..b94e3ef
--- /dev/null
+++ b/gotfparse/cmd/tftest/README.md
@@ -0,0 +1,5 @@
+# tftest
+
+`tftest` is used when developing to allow for rapid iteration of the gotfparse library.
+
+    go run cmd/tftest/main.go <path-to-terraform> > output.json
\ No newline at end of file
diff --git a/gotfparse/cmd/tftest/main.go b/gotfparse/cmd/tftest/main.go
new file mode 100644
index 0000000..cf8e120
--- /dev/null
+++ b/gotfparse/cmd/tftest/main.go
@@ -0,0 +1,32 @@
+// Copyright The Cloud Custodian Authors.
+// SPDX-License-Identifier: Apache-2.0
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"log"
+	"os"
+
+	"github.com/cloud-custodian/tfparse/gotfparse/pkg/converter"
+)
+
+func main() {
+	if len(os.Args) < 2 || len(os.Args) > 2 {
+		log.Fatal("expected 1 argument, path")
+	}
+
+	path := os.Args[1]
+	tfd, err := converter.NewTerraformConverter(path)
+	if err != nil {
+		fmt.Print(err)
+		return
+	}
+	data := tfd.VisitJSON().Data()
+	j, err := json.MarshalIndent(data, "", "\t")
+	if err != nil {
+		fmt.Print(err)
+		return
+	}
+	fmt.Print(string(j))
+}