Repository: bacalhau-project/bacalhau (language: python)

Issue:
Remove code referencing estuary 
Estuary was decommissioned in July 2023. Should be save to remove from the codebase now.


Discussion / hints:

Fix patch:
diff --git a/docs/docs/all-flags.md b/docs/docs/all-flags.md
index 425a45fce3..1f9614b554 100644
--- a/docs/docs/all-flags.md
+++ b/docs/docs/all-flags.md
@@ -287,7 +287,7 @@ Flags:
       --node-details                     Print out details of all nodes (overridden by --id-only).
       --output-dir string                Directory to write the output to.
   -o, --output-volumes strings           name:path of the output data volumes. 'outputs:/outputs' is always added.
-  -p, --publisher publisher              Where to publish the result of the job (default Estuary)
+  -p, --publisher publisher              Where to publish the result of the job (default IPFS)
       --raw                              Download raw result CIDs instead of merging multiple CIDs into a single result
   -s, --selector string                  Selector (label query) to filter nodes on which this job can be executed, supports '=', '==', and '!='.(e.g. -s key1=value1,key2=value2). Matching objects must satisfy all of the specified label constraints.
       --skip-syntax-checking             Skip having 'shellchecker' verify syntax of the command
@@ -459,7 +459,6 @@ Examples:
   bacalhau serve --peer env --private-internal-ipfs=false
 
 Flags:
-      --estuary-api-key string                           The API key used when using the estuary API.
       --filecoin-unsealed-path string                    The go template that can turn a filecoin CID into a local filepath with the unsealed data.
   -h, --help                                             help for serve
       --host string                                      The host to listen on (for both api and swarm connections). (default "0.0.0.0")
diff --git a/docs/docs/data-ingestion/from-url.md b/docs/docs/data-ingestion/from-url.md
index 90c9ed7003..eda0896d8d 100644
--- a/docs/docs/data-ingestion/from-url.md
+++ b/docs/docs/data-ingestion/from-url.md
@@ -27,7 +27,7 @@ Let's look closely at the command above:
 
 * ` --input=https://raw.githubusercontent.com/filecoin-project/bacalhau/main/README.md \`: URL path of the input data volumes downloaded from a URL source. 
 
-The `bacalhau docker run` command takes advantage of the `--input` parameter. This will download a file from a public URL and place it in the `/inputs` directory of the container (by default). Then we will use a helper container to move that data to the `/outputs` directory so that it is published to your public storage via Estuary. In our case we are using Filecoin as our public storage.
+The `bacalhau docker run` command takes advantage of the `--input` parameter. This will download a file from a public URL and place it in the `/inputs` directory of the container (by default). Then we will use a helper container to move that data to the `/outputs` directory so that it is published to your public storage via IPFS. In our case we are using Filecoin as our public storage.
 
 :::tip
 You can find out more about the [helper container in the examples repository](https://github.com/bacalhau-project/examples/tree/main/tools/upload).
@@ -76,7 +76,7 @@ To get the output CID from a completed job, run the following command:
 %%bash --out cid
 bacalhau list $JOB_ID --output=json | jq -r '.[0].Status.JobState.Nodes[] | .Shards."0".PublishedResults | select(.CID) | .CID'
 ```
-The job will upload the CID to your public storage via Estuary. We will store the _cid_ that in an environment variable so that we can reuse it later on.
+The job will upload the CID to your public storage via IPFS. We will store the _cid_ that in an environment variable so that we can reuse it later on.
 
 ### Use the CID in a New Bacalhau Job
 
diff --git a/docs/docs/data-ingestion/pin.md b/docs/docs/data-ingestion/pin.md
index 0da7cb71a8..3db2b8b511 100644
--- a/docs/docs/data-ingestion/pin.md
+++ b/docs/docs/data-ingestion/pin.md
@@ -5,7 +5,7 @@ description: "How to pin data to public storage"
 ---
 # Pinning Data
 
-If you have data that you want to make available to your Bacalhau jobs (or other people), you can pin it using a pinning service like Web3.Storage, Estuary, etc. Pinning services store data on behalf of users. The pinning provider is essentially guaranteeing that your data will be available if someone knows the CID. Most pinning services offer you a free tier, so you can try them out without spending any money.
+If you have data that you want to make available to your Bacalhau jobs (or other people), you can pin it using a pinning service like Web3.Storage, Pinata, etc. Pinning services store data on behalf of users. The pinning provider is essentially guaranteeing that your data will be available if someone knows the CID. Most pinning services offer you a free tier, so you can try them out without spending any money.
 
 ## Web3.Storage
 
@@ -65,36 +65,3 @@ curl -X POST https://api.web3.storage/upload -H "Authorization: Bearer ${TOKEN}"
 docker run --rm --env TOKEN=$TOKEN -v $PWD/nodejs:/nodejs node:18-alpine ash -c 'cd /nodejs && wget http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz && npm install && node put-files.js --token=$TOKEN train-images-idx3-ubyte.gz'
 ```
 
-## Estuary
-
-This example shows you how to pin data using [estuary](https://estuary.tech/api-admin).
-
-- Before you can upload files via estuary, you must create an [account](https://estuary.tech) (if you don't have one already).
-
-- Browse to [the API Key management page](https://estuary.tech/api-admin) and create a key.
-
-### Ways to pin using Esturay
-
-1. **Pin a local file via the Esturay UI**: You can [browse to the Estuary UI](https://estuary.tech/upload) to upload a file via your web browser.
-
-:::tip
-Due to the way Estuary batches files for pinning, it may take some time before your file is accessible/listable.
-:::
-
-2. **Pin a local file via Curl**: Please view the [API documentation](https://docs.estuary.tech/tutorial-uploading-your-first-file) to see all available commands. This example submits a single file to be pinned.
-
-```bash
-export TOKEN=YOUR_API_KEY
-echo hello world > foo.txt
-curl -X POST https://upload.estuary.tech/content/add -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: multipart/form-data" -F "data=@foo.txt"
-```
-
-The response will return the CID of the file.
-
-## View pinned files
-
-If the upload was successful, you can view the file via your [estuary account page](https://estuary.tech/home). Alternatively, you can obtain this information from the CLI:
-
-```bash
-curl -X GET -H "Authorization: Bearer ${TOKEN}" https://api.estuary.tech/content/list
-```
diff --git a/docs/docs/integration/lilypad.md b/docs/docs/integration/lilypad.md
index 52bea0fa97..8b743e8e71 100644
--- a/docs/docs/integration/lilypad.md
+++ b/docs/docs/integration/lilypad.md
@@ -146,7 +146,7 @@ Here's an example JSON job specification for the Stable Diffusion job:
 {
   "Engine": "docker",
   "Verifier": "noop",
-  "PublisherSpec": { "Type": "estuary" },
+  "PublisherSpec": { "Type": "ipfs" },
   "Docker": {
     "Image": "ghcr.io/bacalhau-project/examples/stable-diffusion-gpu:0.0.1",
     "Entrypoint": ["python"],
@@ -172,7 +172,7 @@ Note that since we need to be able to add the user prompt input to the spec, it'
 string constant specStart = '{'
     '"Engine": "docker",'
     '"Verifier": "noop",'
-    '"PublisherSpec": {"Type": "estuary"},'
+    '"PublisherSpec": {"Type": "ipfs"},'
     '"Docker": {'
     '"Image": "ghcr.io/bacalhau-project/examples/stable-diffusion-gpu:0.0.1",'
     '"Entrypoint": ["python"],
@@ -264,7 +264,7 @@ contract MyContract is LilypadCallerInterface {
   string constant specStart = '{'
       '"Engine": "docker",'
       '"Verifier": "noop",'
-      '"PublisherSpec": {"Type": "estuary"},'
+      '"PublisherSpec": {"Type": "ipfs"},'
       '"Docker": {'
       '"Image": "ghcr.io/bacalhau-project/examples/stable-diffusion-gpu:0.0.1",'
       '"Entrypoint": ["python"],
diff --git a/docs/docs/references/sdk/python-sdk.md b/docs/docs/references/sdk/python-sdk.md
index 822e76f975..06091de717 100644
--- a/docs/docs/references/sdk/python-sdk.md
+++ b/docs/docs/references/sdk/python-sdk.md
@@ -78,7 +78,7 @@ data = dict(
     Spec=Spec(
         engine="Docker",
         verifier="Noop",
-        publisher_spec=PublisherSpec(type="Estuary"),
+        publisher_spec=PublisherSpec(type="IPFS"),
         docker=JobSpecDocker(
             image="ubuntu",
             entrypoint=["echo", "Hello World!"],
@@ -121,7 +121,7 @@ The script above prints the following object, the `job.metadata.id` value is our
                   'ExecutionPlan': {'ShardsTotal': 1},
                   'Language': {'JobContext': {}},
                   'Network': {'Type': 'None'},
-                  'Publisher': 'Estuary',
+                  'Publisher': 'IPFS',
                   'Resources': {'GPU': ''},
                   'Sharding': {'BatchSize': 1,
                                'GlobPatternBasePath': '/inputs'},
diff --git a/docs/swagger.json b/docs/swagger.json
index 1dbc087239..3332943aed 100644
--- a/docs/swagger.json
+++ b/docs/swagger.json
@@ -1608,7 +1608,6 @@
                 "publisherUnknown",
                 "PublisherNoop",
                 "PublisherIpfs",
-                "PublisherEstuary",
                 "PublisherS3",
                 "publisherDone"
             ]
@@ -1846,7 +1845,6 @@
                 "StorageSourceRepoClone",
                 "StorageSourceRepoCloneLFS",
                 "StorageSourceURLDownload",
-                "StorageSourceEstuary",
                 "StorageSourceInline",
                 "StorageSourceLocalDirectory",
                 "StorageSourceS3",
@@ -2157,4 +2155,4 @@
             }
         }
     }
-}
+}
\ No newline at end of file
diff --git a/integration/airflow/docs/index.rst b/integration/airflow/docs/index.rst
index 5f802c0e51..9ace66b5f5 100644
--- a/integration/airflow/docs/index.rst
+++ b/integration/airflow/docs/index.rst
@@ -43,7 +43,7 @@ This package contains the python Bacalhau Airflow Provider named `bacalhau_airfl
          api_version="V1beta1",
          job_spec=dict(
                engine="Docker",
-               publisher="Estuary",
+               publisher="IPFS",
                docker=dict(
                   image="ubuntu",
                   entrypoint=["echo", "World"],
diff --git a/pkg/config/configenv/dev.go b/pkg/config/configenv/dev.go
index 99d85eec14..ebca3f2824 100644
--- a/pkg/config/configenv/dev.go
+++ b/pkg/config/configenv/dev.go
@@ -40,7 +40,6 @@ var Development = types.BacalhauConfig{
 		DownloadURLRequestRetries: 3,
 		LoggingMode:               logger.LogModeDefault,
 		Type:                      []string{"requester"},
-		EstuaryAPIKey:             "",
 		AllowListedLocalPaths:     []string{},
 		Labels:                    map[string]string{},
 		DisabledFeatures: types.FeatureConfig{
diff --git a/pkg/config/configenv/local.go b/pkg/config/configenv/local.go
index 1563673c59..8b8f50ad83 100644
--- a/pkg/config/configenv/local.go
+++ b/pkg/config/configenv/local.go
@@ -37,7 +37,6 @@ var Local = types.BacalhauConfig{
 		DownloadURLRequestRetries: 3,
 		LoggingMode:               logger.LogModeDefault,
 		Type:                      []string{"requester"},
-		EstuaryAPIKey:             "",
 		AllowListedLocalPaths:     []string{},
 		Labels:                    map[string]string{},
 		DisabledFeatures: types.FeatureConfig{
diff --git a/pkg/config/configenv/production.go b/pkg/config/configenv/production.go
index 836b39a028..604f22b1bc 100644
--- a/pkg/config/configenv/production.go
+++ b/pkg/config/configenv/production.go
@@ -42,7 +42,6 @@ var Production = types.BacalhauConfig{
 		DownloadURLRequestRetries: 3,
 		LoggingMode:               logger.LogModeDefault,
 		Type:                      []string{"requester"},
-		EstuaryAPIKey:             "",
 		AllowListedLocalPaths:     []string{},
 		Labels:                    map[string]string{},
 		DisabledFeatures: types.FeatureConfig{
diff --git a/pkg/config/configenv/staging.go b/pkg/config/configenv/staging.go
index ee3f510219..3a2d05b020 100644
--- a/pkg/config/configenv/staging.go
+++ b/pkg/config/configenv/staging.go
@@ -42,7 +42,6 @@ var Staging = types.BacalhauConfig{
 		DownloadURLRequestRetries: 3,
 		LoggingMode:               logger.LogModeDefault,
 		Type:                      []string{"requester"},
-		EstuaryAPIKey:             "",
 		AllowListedLocalPaths:     []string{},
 		Labels:                    map[string]string{},
 		DisabledFeatures: types.FeatureConfig{
diff --git a/pkg/config/types/generated_constants.go b/pkg/config/types/generated_constants.go
index 08c0fa0f6b..2b13b91540 100644
--- a/pkg/config/types/generated_constants.go
+++ b/pkg/config/types/generated_constants.go
@@ -123,7 +123,6 @@ const NodeExecutorPluginPath = "Node.ExecutorPluginPath"
 const NodeComputeStoragePath = "Node.ComputeStoragePath"
 const NodeLoggingMode = "Node.LoggingMode"
 const NodeType = "Node.Type"
-const NodeEstuaryAPIKey = "Node.EstuaryAPIKey"
 const NodeAllowListedLocalPaths = "Node.AllowListedLocalPaths"
 const NodeDisabledFeatures = "Node.DisabledFeatures"
 const NodeDisabledFeaturesEngines = "Node.DisabledFeatures.Engines"
diff --git a/pkg/config/types/generated_viper_defaults.go b/pkg/config/types/generated_viper_defaults.go
index 0c796da28d..90e7b1fa74 100644
--- a/pkg/config/types/generated_viper_defaults.go
+++ b/pkg/config/types/generated_viper_defaults.go
@@ -4,7 +4,6 @@ package types
 
 import "github.com/spf13/viper"
 
-
 type SetOption func(p *SetParams)
 
 func WithViper(v *viper.Viper) SetOption {
@@ -138,7 +137,6 @@ func SetDefaults(cfg BacalhauConfig, opts ...SetOption) {
 	p.Viper.SetDefault(NodeComputeStoragePath, cfg.Node.ComputeStoragePath)
 	p.Viper.SetDefault(NodeLoggingMode, cfg.Node.LoggingMode)
 	p.Viper.SetDefault(NodeType, cfg.Node.Type)
-	p.Viper.SetDefault(NodeEstuaryAPIKey, cfg.Node.EstuaryAPIKey)
 	p.Viper.SetDefault(NodeAllowListedLocalPaths, cfg.Node.AllowListedLocalPaths)
 	p.Viper.SetDefault(NodeDisabledFeatures, cfg.Node.DisabledFeatures)
 	p.Viper.SetDefault(NodeDisabledFeaturesEngines, cfg.Node.DisabledFeatures.Engines)
@@ -284,7 +282,6 @@ func Set(cfg BacalhauConfig, opts ...SetOption) {
 	p.Viper.Set(NodeComputeStoragePath, cfg.Node.ComputeStoragePath)
 	p.Viper.Set(NodeLoggingMode, cfg.Node.LoggingMode)
 	p.Viper.Set(NodeType, cfg.Node.Type)
-	p.Viper.Set(NodeEstuaryAPIKey, cfg.Node.EstuaryAPIKey)
 	p.Viper.Set(NodeAllowListedLocalPaths, cfg.Node.AllowListedLocalPaths)
 	p.Viper.Set(NodeDisabledFeatures, cfg.Node.DisabledFeatures)
 	p.Viper.Set(NodeDisabledFeaturesEngines, cfg.Node.DisabledFeatures.Engines)
diff --git a/pkg/config/types/node.go b/pkg/config/types/node.go
index 47c6193f10..b06071beb4 100644
--- a/pkg/config/types/node.go
+++ b/pkg/config/types/node.go
@@ -32,8 +32,6 @@ type NodeConfig struct {
 	LoggingMode logger.LogMode `yaml:"LoggingMode"`
 	// Type is "compute", "requester" or both
 	Type []string `yaml:"Type"`
-	// Deprecated: TODO(forrest) remove.
-	EstuaryAPIKey string `yaml:"EstuaryAPIKey"`
 	// Local paths that are allowed to be mounted into jobs
 	AllowListedLocalPaths []string `yaml:"AllowListedLocalPaths"`
 	// What features should not be enabled even if installed
diff --git a/pkg/job/parser.go b/pkg/job/parser.go
index b5715ddd50..729f01b0b9 100644
--- a/pkg/job/parser.go
+++ b/pkg/job/parser.go
@@ -122,7 +122,7 @@ func ParsePublisherString(destinationURI string, options map[string]interface{})
 
 	var res model.PublisherSpec
 	switch parsedURI.Scheme {
-	case "ipfs", "estuary": // Also handle the deprecated estuary publisher
+	case "ipfs":
 		res = model.PublisherSpec{
 			Type: model.PublisherIpfs,
 		}
diff --git a/pkg/model/publisher.go b/pkg/model/publisher.go
index 58c24d1f8e..ddaf3b7238 100644
--- a/pkg/model/publisher.go
+++ b/pkg/model/publisher.go
@@ -10,16 +10,14 @@ const (
 	publisherUnknown Publisher = iota // must be first
 	PublisherNoop
 	PublisherIpfs
-	PublisherEstuary
 	PublisherS3
 	publisherDone // must be last
 )
 
 var publisherNames = map[Publisher]string{
-	PublisherNoop:    "noop",
-	PublisherIpfs:    "ipfs",
-	PublisherEstuary: "estuary",
-	PublisherS3:      "s3",
+	PublisherNoop: "noop",
+	PublisherIpfs: "ipfs",
+	PublisherS3:   "s3",
 }
 
 func ParsePublisher(str string) (Publisher, error) {
diff --git a/pkg/model/storage_source.go b/pkg/model/storage_source.go
index 4c05f55b07..bfce60d452 100644
--- a/pkg/model/storage_source.go
+++ b/pkg/model/storage_source.go
@@ -16,7 +16,6 @@ const (
 	StorageSourceRepoClone
 	StorageSourceRepoCloneLFS
 	StorageSourceURLDownload
-	StorageSourceEstuary
 	StorageSourceInline
 	StorageSourceLocalDirectory
 	StorageSourceS3
@@ -28,7 +27,6 @@ var storageSourceNames = map[StorageSourceType]string{
 	StorageSourceRepoClone:      "repoClone",
 	StorageSourceRepoCloneLFS:   "repoCloneLFS",
 	StorageSourceURLDownload:    "urlDownload",
-	StorageSourceEstuary:        "estuary",
 	StorageSourceInline:         "inline",
 	StorageSourceLocalDirectory: "localDirectory",
 	StorageSourceS3:             "s3",
diff --git a/pkg/swagger/docs.go b/pkg/swagger/docs.go
index d0a9448026..309ddea4cc 100644
--- a/pkg/swagger/docs.go
+++ b/pkg/swagger/docs.go
@@ -2071,7 +2071,6 @@ const docTemplate = `{
                 "publisherUnknown",
                 "PublisherNoop",
                 "PublisherIpfs",
-                "PublisherEstuary",
                 "PublisherS3",
                 "publisherDone"
             ]
@@ -2309,7 +2308,6 @@ const docTemplate = `{
                 "StorageSourceRepoClone",
                 "StorageSourceRepoCloneLFS",
                 "StorageSourceURLDownload",
-                "StorageSourceEstuary",
                 "StorageSourceInline",
                 "StorageSourceLocalDirectory",
                 "StorageSourceS3",
diff --git a/webui/public/swagger/swagger.json b/webui/public/swagger/swagger.json
index 852a41c257..e34049a157 100644
--- a/webui/public/swagger/swagger.json
+++ b/webui/public/swagger/swagger.json
@@ -2067,7 +2067,6 @@
                 "publisherUnknown",
                 "PublisherNoop",
                 "PublisherIpfs",
-                "PublisherEstuary",
                 "PublisherS3",
                 "publisherDone"
             ]
@@ -2305,7 +2304,6 @@
                 "StorageSourceRepoClone",
                 "StorageSourceRepoCloneLFS",
                 "StorageSourceURLDownload",
-                "StorageSourceEstuary",
                 "StorageSourceInline",
                 "StorageSourceLocalDirectory",
                 "StorageSourceS3",


Test patch:
diff --git a/integration/airflow/tests/test_hooks.py b/integration/airflow/tests/test_hooks.py
index d97283d542..7865526060 100644
--- a/integration/airflow/tests/test_hooks.py
+++ b/integration/airflow/tests/test_hooks.py
@@ -22,7 +22,7 @@ def test_submit_job(self):
         api_version = "V1beta1"
         spec = dict(
             engine="Docker",
-            publisher="Estuary",
+            publisher="IPFS",
             docker=dict(
                 image="ubuntu",
                 entrypoint=["echo", "TestBacalhauAirflowOperator"],
diff --git a/integration/airflow/tests/test_operators.py b/integration/airflow/tests/test_operators.py
index 8761a3e287..0d3efb7a3d 100644
--- a/integration/airflow/tests/test_operators.py
+++ b/integration/airflow/tests/test_operators.py
@@ -32,7 +32,7 @@ def setUp(self):
             api_version="V1beta1",
             job_spec=dict(
                 engine="Docker",
-                publisher="Estuary",
+                publisher="IPFS",
                 docker=dict(
                     image="ubuntu",
                     entrypoint=["echo", "TestBacalhauSubmitJobOperator"],
diff --git a/pkg/config/configenv/test.go b/pkg/config/configenv/test.go
index af454128eb..b5611f3831 100644
--- a/pkg/config/configenv/test.go
+++ b/pkg/config/configenv/test.go
@@ -39,7 +39,6 @@ var Testing = types.BacalhauConfig{
 		DownloadURLRequestRetries: 3,
 		LoggingMode:               logger.LogModeDefault,
 		Type:                      []string{"requester"},
-		EstuaryAPIKey:             "",
 		AllowListedLocalPaths:     []string{},
 		Labels:                    map[string]string{},
 		DisabledFeatures: types.FeatureConfig{
diff --git a/testdata/job-test.yaml b/testdata/job-test.yaml
index d96f635fe2..18ecc43d17 100644
--- a/testdata/job-test.yaml
+++ b/testdata/job-test.yaml
@@ -2,23 +2,23 @@
 APIVersion: v1beta1
 Engine: Docker
 Verifier: Noop
-Publisher: Estuary
+Publisher: IPFS
 JobSpecDocker:
   # docker image you want to use
   Image: ubuntu
   Entrypoint:
-    # Command to run inside the container
-    - /bin/bash
-    - -c
-    - echo "hello"
+  # Command to run inside the container
+  - /bin/bash
+  - -c
+  - echo "hello"
   EnvironmentVariables: []
 # if your job consumes data from ipfs or from a url since in this job we don't consume any data it's left empty
 inputs: []
 Contexts: []
 outputs:
-  - StorageEngine: IPFS
-    # default output volume is outputs
-    Name: outputs
-    CID: ""
-    URL: ""
-    path: /outputs
+- StorageEngine: IPFS
+  # default output volume is outputs
+  Name: outputs
+  CID: ""
+  URL: ""
+  path: /outputs