Repository: stellar/go
Pull request #5335 (2024-06-07 03:45:47)

exp/services/ledgerexporter: refactor ledgerexporter cli into sub-commands
<!-- If you're making a doc PR or something tiny where the below is irrelevant, delete this
template and use a short description, but in your description aim to include both what the
change is, and why it is being made, with enough context for anyone to understand. -->

<details>
  <summary>PR Checklist</summary>
  
### PR Structure

* [x] This PR has reasonably narrow scope (if not, break it down into smaller PRs).
* [x] This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
* [x] This PR's title starts with name of package that is most changed in the PR, ex.
  `services/friendbot`, or `all` or `doc` if the changes are broad or impact many
  packages.

### Thoroughness

* [x] This PR adds tests for the most critical parts of the new functionality or fixes.
* [ ] I've updated any docs ([developer docs](https://developers.stellar.org/api/), `.md`
  files, etc... affected by this change). Take a look in the `docs` folder for a given service,
  like [this one](https://github.com/stellar/go/tree/master/services/horizon/internal/docs).

### Release planning

* [ ] I've updated the relevant CHANGELOG ([here](services/horizon/CHANGELOG.md) for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
* [ ] I've decided if this PR requires a new major/minor version according to
  [semver](https://semver.org/), or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.
</details>

### What

* changed the cli interface to accept new sub-commands and removed --resume flag:

  `ledgerexporter scan-and-fill --start --end --config-file` - exports the entire bounded requested range, only the ledgers which are missing are uploaded to the datastore.

  `ledgerexporter append --start --end --config-file` - export ledgers beginning with dynamically determined starting point based on binary search across `--start` and `--end` for the first absent ledger with that range.

* updated config settings now allow for disabling testnet/pubnet preconfigured settings for captive core and providing a custom external captive core config file instead via `stellar_core_config.captive_core_toml_path`

* this also includes a breaking change on toml config:
  ```
  network_name = "testnet"  
  
  [datastore_config]
  type = "GCS"
  
  [datastore_config.params]
  destination_bucket_path = "your-bucket-name/<optional_subpaths>"
  
  [datastore_config.schema]
  ledgers_per_file = 64
  files_per_partition = 10
  file_suffix = ".xdr.gz"
  
  # preconfigured captive config
  [stellar_core_config]
    stellar_core_binary_path = "/my/path/to/stellar-core"
    preconfigured_network = "testnet" # Options: `testnet` or `pubnet`
  
  or
  
  # manual  captive config
  [stellar_core_config]
    captive_core_toml_path = "my-captive-core.cfg"
    stellar_core_binary_path = "/my/path/to/stellar-core"
    history_archive_urls = ["http://testarchiveurl1", "http://testarchiveurl2"]
    network_passphrase = "test"
  
  ```

### Why

* sub-commands conveniently encapsulate lower level aspects of export outcomes and performance to make it best outcome for user automatically, such as using 'resume'. 

* operators needed ability to override the preconfigured testnet/pubnet captive core settings.

Closes [hubble-271](https://stellarorg.atlassian.net/browse/HUBBLE-271)

### Known limitations





Patch:
diff --git a/exp/services/ledgerexporter/README.md b/exp/services/ledgerexporter/README.md
index 967bd1dc..57757508 100644
--- a/exp/services/ledgerexporter/README.md
+++ b/exp/services/ledgerexporter/README.md
@@ -21,45 +21,56 @@ type LedgerCloseMetaBatch struct {
 
 ### Command Line Options
 
-#### Bounded Mode:
-Exports a specific range of ledgers, defined by --start and --end.
+#### Scan and Fill Mode:
+Exports a specific range of ledgers, defined by --start and --end. Will only export to remote datastore if data is absent.
 ```bash
-ledgerexporter --start <start_ledger> --end <end_ledger> --config-file <config_file_path>
+ledgerexporter scan-and-fill --start <start_ledger> --end <end_ledger> --config-file <config_file_path>
 ```
 
-#### Unbounded Mode:
-Exports ledgers continuously starting from --start. In this mode, the end ledger is either not provided or set to 0.
+#### Append Mode:
+Exports ledgers initially searching from --start, looking for the next absent ledger sequence number proceeding --start on the data store. If abscence is detected, the export range is narrowed to `--start <absent_ledger_sequence>`. 
+This feature requires ledgers to be present on the remote data store for some (possibly empty) prefix of the requested range and then absent for the (possibly empty) remainder. 
+
+In this mode, the --end ledger can be provided to stop the process once export has reached that ledger, or if absent or 0 it will result in continous exporting of new ledgers emitted from the network. 
+
+ It’s guaranteed that ledgers exported during `append` mode from `start` and up to the last logged ledger file `Uploaded {ledger file name}` were contiguous, meaning all ledgers within that range were exported to the data lake with no gaps or missing ledgers in between.
 ```bash
-ledgerexporter --start <start_ledger> --config-file <config_file_path>
+ledgerexporter append --start <start_ledger> --config-file <config_file_path>
 ```
 
-#### Resumability:
-Exporting a ledger range can be optimized further by enabling resumability if the remote data store supports it.
+### Configuration (toml):
+The `stellar_core_config` supports two ways for configuring captive core:
+  - use prebuilt captive core config toml, archive urls, and passphrase based on `stellar_core_config.network = testnet|pubnet`.
+  - manually set the the captive core confg by supplying these core parameters which will override any defaults when `stellar_core_config.network` is present also:
+    `stellar_core_config.captive_core_toml_path`
+    `stellar_core_config.history_archive_urls`
+    `stellar_core_config.network_passphrase`
 
-By default, resumability is disabled, `--resume false`
+Ensure you have stellar-core installed and set `stellar_core_config.stellar_core_binary_path` to it's path on o/s.
 
-When enabled, `--resume true`, ledgerexporter will search the remote data store within the requested range, looking for the oldest absent ledger sequence number within range. If abscence is detected, the export range is narrowed to `--start <absent_ledger_sequence>`. 
-This feature requires all ledgers to be present on the remote data store for some (possibly empty) prefix of the requested range and then absent for the (possibly empty) remainder.
-
-### Configuration (toml):
+Enable web service that will be bound to localhost post and publishes metrics by including `admin_port = {port}`
 
+An example config, demonstrating preconfigured captive core settings and gcs data store config.
 ```toml
-network = "testnet"  # Options: `testnet` or `pubnet`
+admin_port = 6061
 
 [datastore_config]
 type = "GCS"
 
 [datastore_config.params]
-destination_bucket_path = "your-bucket-name/<optional_subpaths>"
+destination_bucket_path = "your-bucket-name/<optional_subpath1>/<optional_subpath2>/"
 
-[exporter_config]
+[datastore_config.schema]
 ledgers_per_file = 64
 files_per_partition = 10
-```
 
-#### Stellar-core configuration:
-- The exporter automatically configures stellar-core based on the network specified in the config.
-- Ensure you have stellar-core installed and accessible in your system's $PATH.
+[stellar_core_config]
+  network = "testnet"
+  stellar_core_binary_path = "/my/path/to/stellar-core"
+  captive_core_toml_path = "my-captive-core.cfg"
+  history_archive_urls = ["http://testarchiveurl1", "http://testarchiveurl2"]
+  network_passphrase = "test"
+```
 
 ### Exported Files
 
diff --git a/exp/services/ledgerexporter/config.toml b/exp/services/ledgerexporter/config.toml
index a5608742..c41d9376 100644
--- a/exp/services/ledgerexporter/config.toml
+++ b/exp/services/ledgerexporter/config.toml
@@ -1,15 +1,14 @@
-network = "testnet"
-
 [datastore_config]
 type = "GCS"
 
 [datastore_config.params]
-destination_bucket_path = "exporter-test/ledgers"
+destination_bucket_path = "exporter-test/ledgers/testnet"
 
-[exporter_config]
-  ledgers_per_file = 1
-  files_per_partition = 64000
+[datastore_config.schema]
+ledgers_per_file = 1
+files_per_partition = 64000
 
 [stellar_core_config]
+  network = "testnet"
   stellar_core_binary_path = "/usr/local/bin/stellar-core"
 
diff --git a/exp/services/ledgerexporter/docker/start b/exp/services/ledgerexporter/docker/start
index eca213de..3b61c74e 100644
--- a/exp/services/ledgerexporter/docker/start
+++ b/exp/services/ledgerexporter/docker/start
@@ -17,23 +17,25 @@ files_per_partition="${FILES_PER_PARTITION:-64000}"
 
 # Generate TOML configuration
 cat <<EOF > config.toml
-network = "${NETWORK}"
 
 [datastore_config]
 type = "GCS"
 
 [datastore_config.params]
-destination_bucket_path = "${ARCHIVE_TARGET}"
+destination_bucket_path = "${ARCHIVE_TARGET}/${NETWORK}"
 
-[exporter_config]
+[datastore_config.schema]
   ledgers_per_file = $ledgers_per_file
   files_per_partition = $files_per_partition
+
+[stellar_core_config]
+  network = "${NETWORK}"
 EOF
 
 # Check if START or END variables are set
 if [[ -n "$START" || -n "$END" ]]; then
     echo "START: $START END: $END"
-    /usr/bin/ledgerexporter --config-file config.toml --start $START --end $END
+    /usr/bin/ledgerexporter scan-and-fill --config-file config.toml --start $START --end $END
 else
     echo "Error: No ledger range provided."
     exit 1
diff --git a/exp/services/ledgerexporter/internal/app.go b/exp/services/ledgerexporter/internal/app.go
index c6adcd41..eb0f544e 100644
--- a/exp/services/ledgerexporter/internal/app.go
+++ b/exp/services/ledgerexporter/internal/app.go
@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"net/http"
 	"os"
+	"os/exec"
 	"os/signal"
 	"sync"
 	"syscall"
@@ -72,7 +73,7 @@ type InvalidDataStoreError struct {
 func (m InvalidDataStoreError) Error() string {
 	return fmt.Sprintf("The remote data store has inconsistent data, "+
 		"a resumable starting ledger of %v was identified, "+
-		"but that is not aligned to expected ledgers-per-file of %v. use '--resume false' to bypass",
+		"but that is not aligned to expected ledgers-per-file of %v. use 'scan-and-fill' sub-command to bypass",
 		m.LedgerSequence, m.LedgersPerFile)
 }
 
@@ -82,17 +83,16 @@ type App struct {
 	dataStore     datastore.DataStore
 	exportManager *ExportManager
 	uploader      Uploader
-	flags         Flags
 	adminServer   *http.Server
 }
 
-func NewApp(flags Flags) *App {
+func NewApp() *App {
 	logger.SetLevel(log.DebugLevel)
-	app := &App{flags: flags}
+	app := &App{}
 	return app
 }
 
-func (a *App) init(ctx context.Context) error {
+func (a *App) init(ctx context.Context, runtimeSettings RuntimeSettings) error {
 	var err error
 	var archive historyarchive.ArchiveInterface
 
@@ -104,20 +104,22 @@ func (a *App) init(ctx context.Context) error {
 		collectors.NewGoCollector(),
 	)
 
-	if a.config, err = NewConfig(a.flags); err != nil {
+	if a.config, err = NewConfig(runtimeSettings); err != nil {
 		return errors.Wrap(err, "Could not load configuration")
 	}
-	if archive, err = datastore.CreateHistoryArchiveFromNetworkName(ctx, a.config.Network); err != nil {
+	if archive, err = a.config.GenerateHistoryArchive(ctx); err != nil {
+		return err
+	}
+	if err = a.config.ValidateAndSetLedgerRange(ctx, archive); err != nil {
 		return err
 	}
-	a.config.ValidateAndSetLedgerRange(ctx, archive)
 
-	if a.dataStore, err = datastore.NewDataStore(ctx, a.config.DataStoreConfig, a.config.Network); err != nil {
+	if a.dataStore, err = datastore.NewDataStore(ctx, a.config.DataStoreConfig); err != nil {
 		return errors.Wrap(err, "Could not connect to destination data store")
 	}
-	if a.config.Resume {
+	if a.config.Resumable() {
 		if err = a.applyResumability(ctx,
-			datastore.NewResumableManager(a.dataStore, a.config.Network, a.config.LedgerBatchConfig, archive)); err != nil {
+			datastore.NewResumableManager(a.dataStore, a.config.DataStoreConfig.Schema, archive)); err != nil {
 			return err
 		}
 	}
@@ -129,7 +131,10 @@ func (a *App) init(ctx context.Context) error {
 	}
 
 	queue := NewUploadQueue(uploadQueueCapacity, registry)
-	if a.exportManager, err = NewExportManager(a.config, a.ledgerBackend, queue, registry); err != nil {
+	if a.exportManager, err = NewExportManager(a.config.DataStoreConfig.Schema,
+		a.ledgerBackend, queue, registry,
+		a.config.StellarCoreConfig.NetworkPassphrase,
+		a.config.CoreVersion); err != nil {
 		return err
 	}
 	a.uploader = NewUploader(a.dataStore, queue, registry)
@@ -151,8 +156,8 @@ func (a *App) applyResumability(ctx context.Context, resumableManager datastore.
 
 	// TODO - evaluate a more robust validation of remote data for ledgers-per-file consistency
 	// this assumes ValidateAndSetLedgerRange() has conditioned the a.config.StartLedger to be at least > 1
-	if absentLedger > 2 && absentLedger != a.config.LedgerBatchConfig.GetSequenceNumberStartBoundary(absentLedger) {
-		return NewInvalidDataStoreError(absentLedger, a.config.LedgerBatchConfig.LedgersPerFile)
+	if absentLedger > 2 && absentLedger != a.config.DataStoreConfig.Schema.GetSequenceNumberStartBoundary(absentLedger) {
+		return NewInvalidDataStoreError(absentLedger, a.config.DataStoreConfig.Schema.LedgersPerFile)
 	}
 	logger.Infof("For export ledger range start=%d, end=%d, the remote storage has some of this data already, will resume at later start ledger of %d", a.config.StartLedger, a.config.EndLedger, absentLedger)
 	a.config.StartLedger = absentLedger
@@ -180,18 +185,19 @@ func newAdminServer(adminPort int, prometheusRegistry *prometheus.Registry) *htt
 	}
 }
 
-func (a *App) Run() {
+func (a *App) Run(runtimeSettings RuntimeSettings) error {
 	ctx, cancel := context.WithCancel(context.Background())
 	defer cancel()
 
-	if err := a.init(ctx); err != nil {
-		var dataAlreadyExported DataAlreadyExportedError
+	if err := a.init(ctx, runtimeSettings); err != nil {
+		var dataAlreadyExported *DataAlreadyExportedError
 		if errors.As(err, &dataAlreadyExported) {
 			logger.Info(err.Error())
 			logger.Info("Shutting down ledger-exporter")
-			return
+			return nil
 		}
-		logger.WithError(err).Fatal("Stopping ledger-exporter")
+		logger.WithError(err).Error("Stopping ledger-exporter")
+		return err
 	}
 	defer a.close()
 
@@ -254,12 +260,16 @@ func (a *App) Run() {
 			logger.WithError(err).Warn("error in internalServer.Shutdown")
 		}
 	}
+	return nil
 }
 
 // newLedgerBackend Creates and initializes captive core ledger backend
 // Currently, only supports captive-core as ledger backend
 func newLedgerBackend(config *Config, prometheusRegistry *prometheus.Registry) (ledgerbackend.LedgerBackend, error) {
-	captiveConfig, err := config.generateCaptiveCoreConfig()
+	// best effort check on a core bin available from PATH to provide as default if
+	// no core bin is provided from config.
+	coreBinFromPath, _ := exec.LookPath("stellar-core")
+	captiveConfig, err := config.GenerateCaptiveCoreConfig(coreBinFromPath)
 	if err != nil {
 		return nil, err
 	}
diff --git a/exp/services/ledgerexporter/internal/config.go b/exp/services/ledgerexporter/internal/config.go
index ef062867..ff0ffbe2 100644
--- a/exp/services/ledgerexporter/internal/config.go
+++ b/exp/services/ledgerexporter/internal/config.go
@@ -4,6 +4,7 @@ import (
 	"context"
 	_ "embed"
 	"fmt"
+	"os"
 	"os/exec"
 	"strings"
 
@@ -16,20 +17,42 @@ import (
 	"github.com/stellar/go/support/datastore"
 	"github.com/stellar/go/support/errors"
 	"github.com/stellar/go/support/ordered"
+	"github.com/stellar/go/support/storage"
 )
 
-const Pubnet = "pubnet"
-const Testnet = "testnet"
+const (
+	Pubnet    = "pubnet"
+	Testnet   = "testnet"
+	UserAgent = "ledgerexporter"
+)
+
+type Mode int
+
+const (
+	_        Mode = iota
+	ScanFill Mode = iota
+	Append
+)
+
+func (mode Mode) Name() string {
+	switch mode {
+	case ScanFill:
+		return "Scan and Fill"
+	case Append:
+		return "Append"
+	}
+	return "none"
+}
 
-type Flags struct {
+type RuntimeSettings struct {
 	StartLedger    uint32
 	EndLedger      uint32
 	ConfigFilePath string
-	Resume         bool
-	AdminPort      uint
+	Mode           Mode
 }
 
 type StellarCoreConfig struct {
+	Network               string   `toml:"network"`
 	NetworkPassphrase     string   `toml:"network_passphrase"`
 	HistoryArchiveUrls    []string `toml:"history_archive_urls"`
 	StellarCoreBinaryPath string   `toml:"stellar_core_binary_path"`
@@ -39,115 +62,126 @@ type StellarCoreConfig struct {
 type Config struct {
 	AdminPort int `toml:"admin_port"`
 
-	Network           string                      `toml:"network"`
-	DataStoreConfig   datastore.DataStoreConfig   `toml:"datastore_config"`
-	LedgerBatchConfig datastore.LedgerBatchConfig `toml:"exporter_config"`
-	StellarCoreConfig StellarCoreConfig           `toml:"stellar_core_config"`
+	DataStoreConfig   datastore.DataStoreConfig `toml:"datastore_config"`
+	StellarCoreConfig StellarCoreConfig         `toml:"stellar_core_config"`
+	UserAgent         string                    `toml:"user_agent"`
 
 	StartLedger uint32
 	EndLedger   uint32
-	Resume      bool
+	Mode        Mode
 
-	CoreVersion string
+	CoreVersion               string
+	SerializedCaptiveCoreToml []byte
 }
 
-// This will generate the config based on commandline flags and toml
+// This will generate the config based on settings
 //
-// flags                 - command line flags
+// settings              - requested settings
 //
 // return                - *Config or an error if any range validation failed.
-func NewConfig(flags Flags) (*Config, error) {
+func NewConfig(settings RuntimeSettings) (*Config, error) {
 	config := &Config{}
 
-	config.StartLedger = uint32(flags.StartLedger)
-	config.EndLedger = uint32(flags.EndLedger)
-	config.Resume = flags.Resume
+	config.StartLedger = uint32(settings.StartLedger)
+	config.EndLedger = uint32(settings.EndLedger)
+	config.Mode = settings.Mode
 
-	logger.Infof("Requested ledger range start=%d, end=%d, resume=%v", config.StartLedger, config.EndLedger, config.Resume)
+	logger.Infof("Requested export mode of %v with start=%d, end=%d", settings.Mode.Name(), config.StartLedger, config.EndLedger)
 
 	var err error
-	if err = config.processToml(flags.ConfigFilePath); err != nil {
+	if err = config.processToml(settings.ConfigFilePath); err != nil {
 		return nil, err
 	}
-	logger.Infof("Config: %v", *config)
+	logger.Infof("Network Config Archive URLs: %v", config.StellarCoreConfig.HistoryArchiveUrls)
+	logger.Infof("Network Config Archive Passphrase: %v", config.StellarCoreConfig.NetworkPassphrase)
+	logger.Infof("Network Config Archive Stellar Core Binary Path: %v", config.StellarCoreConfig.StellarCoreBinaryPath)
+	logger.Infof("Network Config Archive Stellar Core Toml Config: %v", string(config.SerializedCaptiveCoreToml))
 
 	return config, nil
 }
 
+func (config *Config) Resumable() bool {
+	return config.Mode == Append
+}
+
 // Validates requested ledger range, and will automatically adjust it
 // to be ledgers-per-file boundary aligned
 func (config *Config) ValidateAndSetLedgerRange(ctx context.Context, archive historyarchive.ArchiveInterface) error {
+
+	if config.StartLedger < 2 {
+		return errors.New("invalid start value, must be greater than one.")
+	}
+
+	if config.Mode == ScanFill && config.EndLedger == 0 {
+		return errors.New("invalid end value, unbounded mode not supported, end must be greater than start.")
+	}
+
+	if config.EndLedger != 0 && config.EndLedger <= config.StartLedger {
+		return errors.New("invalid end value, must be greater than start")
+	}
+
 	latestNetworkLedger, err := datastore.GetLatestLedgerSequenceFromHistoryArchives(archive)
+	latestNetworkLedger = latestNetworkLedger + (datastore.GetHistoryArchivesCheckPointFrequency() * 2)
 
 	if err != nil {
 		return errors.Wrap(err, "Failed to retrieve the latest ledger sequence from history archives.")
 	}
-	logger.Infof("Latest %v ledger sequence was detected as %d", config.Network, latestNetworkLedger)
+	logger.Infof("Latest ledger sequence was detected as %d", latestNetworkLedger)
 
 	if config.StartLedger > latestNetworkLedger {
-		return errors.Errorf("--start %d exceeds latest network ledger %d",
+		return errors.Errorf("start %d exceeds latest network ledger %d",
 			config.StartLedger, latestNetworkLedger)
 	}
 
-	if config.EndLedger != 0 { // Bounded mode
-		if config.EndLedger < config.StartLedger {
-			return errors.New("invalid --end value, must be >= --start")
-		}
-		if config.EndLedger > latestNetworkLedger {
-			return errors.Errorf("--end %d exceeds latest network ledger %d",
-				config.EndLedger, latestNetworkLedger)
-		}
+	if config.EndLedger > latestNetworkLedger {
+		return errors.Errorf("end %d exceeds latest network ledger %d",
+			config.EndLedger, latestNetworkLedger)
 	}
 
 	config.adjustLedgerRange()
 	return nil
 }
 
-func (config *Config) generateCaptiveCoreConfig() (ledgerbackend.CaptiveCoreConfig, error) {
-	coreConfig := &config.StellarCoreConfig
+func (config *Config) GenerateHistoryArchive(ctx context.Context) (historyarchive.ArchiveInterface, error) {
+	return historyarchive.NewArchivePool(config.StellarCoreConfig.HistoryArchiveUrls, historyarchive.ArchiveOptions{
+		ConnectOptions: storage.ConnectOptions{
+			UserAgent: config.UserAgent,
+			Context:   ctx,
+		},
+	})
+}
 
-	// Look for stellar-core binary in $PATH, if not supplied
-	if config.StellarCoreConfig.StellarCoreBinaryPath == "" {
-		var err error
-		if config.StellarCoreConfig.StellarCoreBinaryPath, err = exec.LookPath("stellar-core"); err != nil {
-			return ledgerbackend.CaptiveCoreConfig{}, errors.Wrap(err, "Failed to find stellar-core binary")
-		}
-	}
+// coreBinDefaultPath - a default value to use for core binary path on system.
+//
+//	this will be used if StellarCoreConfig.StellarCoreBinaryPath is not specified
+func (config *Config) GenerateCaptiveCoreConfig(coreBinFromPath string) (ledgerbackend.CaptiveCoreConfig, error) {
+	var err error
 
-	if err := config.setCoreVersionInfo(); err != nil {
-		return ledgerbackend.CaptiveCoreConfig{}, fmt.Errorf("failed to set stellar-core version info: %w", err)
+	if config.StellarCoreConfig.StellarCoreBinaryPath == "" && coreBinFromPath == "" {
+		return ledgerbackend.CaptiveCoreConfig{}, errors.New("Invalid captive core config, no stellar-core binary path was provided.")
 	}
 
-	var captiveCoreConfig []byte
-	// Default network config
-	switch config.Network {
-	case Pubnet:
-		coreConfig.NetworkPassphrase = network.PublicNetworkPassphrase
-		coreConfig.HistoryArchiveUrls = network.PublicNetworkhistoryArchiveURLs
-		captiveCoreConfig = ledgerbackend.PubnetDefaultConfig
-
-	case Testnet:
-		coreConfig.NetworkPassphrase = network.TestNetworkPassphrase
-		coreConfig.HistoryArchiveUrls = network.TestNetworkhistoryArchiveURLs
-		captiveCoreConfig = ledgerbackend.TestnetDefaultConfig
+	if config.StellarCoreConfig.StellarCoreBinaryPath == "" {
+		config.StellarCoreConfig.StellarCoreBinaryPath = coreBinFromPath
+	}
 
-	default:
-		logger.Fatalf("Invalid network %s", config.Network)
+	if err = config.setCoreVersionInfo(); err != nil {
+		return ledgerbackend.CaptiveCoreConfig{}, fmt.Errorf("failed to set stellar-core version info: %w", err)
 	}
 
 	params := ledgerbackend.CaptiveCoreTomlParams{
-		NetworkPassphrase:  coreConfig.NetworkPassphrase,
-		HistoryArchiveURLs: coreConfig.HistoryArchiveUrls,
+		NetworkPassphrase:  config.StellarCoreConfig.NetworkPassphrase,
+		HistoryArchiveURLs: config.StellarCoreConfig.HistoryArchiveUrls,
 		UseDB:              true,
 	}
 
-	captiveCoreToml, err := ledgerbackend.NewCaptiveCoreTomlFromData(captiveCoreConfig, params)
+	captiveCoreToml, err := ledgerbackend.NewCaptiveCoreTomlFromData(config.SerializedCaptiveCoreToml, params)
 	if err != nil {
 		return ledgerbackend.CaptiveCoreConfig{}, errors.Wrap(err, "Failed to create captive-core toml")
 	}
 
 	return ledgerbackend.CaptiveCoreConfig{
-		BinaryPath:          coreConfig.StellarCoreBinaryPath,
+		BinaryPath:          config.StellarCoreConfig.StellarCoreBinaryPath,
 		NetworkPassphrase:   params.NetworkPassphrase,
 		HistoryArchiveURLs:  params.HistoryArchiveURLs,
 		CheckpointFrequency: datastore.GetHistoryArchivesCheckPointFrequency(),
@@ -186,34 +220,75 @@ func (config *Config) processToml(tomlPath string) error {
 	// Load config TOML file
 	cfg, err := toml.LoadFile(tomlPath)
 	if err != nil {
-		return err
+		return errors.Wrapf(err, "config file %v was not found", tomlPath)
 	}
 
 	// Unmarshal TOML data into the Config struct
-	if err := cfg.Unmarshal(config); err != nil {
+	if err = cfg.Unmarshal(config); err != nil {
 		return errors.Wrap(err, "Error unmarshalling TOML config.")
 	}
 
-	// validate TOML data
-	if config.Network == "" {
-		return errors.New("Invalid TOML config, 'network' must be set, supported values are 'testnet' or 'pubnet'")
+	if config.UserAgent == "" {
+		config.UserAgent = UserAgent
 	}
+
+	if config.StellarCoreConfig.Network == "" && (len(config.StellarCoreConfig.HistoryArchiveUrls) == 0 || config.StellarCoreConfig.NetworkPassphrase == "" || config.StellarCoreConfig.CaptiveCoreTomlPath == "") {
+		return errors.New("Invalid captive core config, the 'network' parameter must be set to pubnet or testnet or " +
+			"'stellar_core_config.history_archive_urls' and 'stellar_core_config.network_passphrase' and 'stellar_core_config.captive_core_toml_path' must be set.")
+	}
+
+	// network config values are an overlay, with network preconfigured values being first if network is present
+	// and then toml settings specific for passphrase, archiveurls, core toml file can override lastly.
+	var networkPassPhrase string
+	var networkArchiveUrls []string
+	switch config.StellarCoreConfig.Network {
+	case "":
+
+	case Pubnet:
+		networkPassPhrase = network.PublicNetworkPassphrase
+		networkArchiveUrls = network.PublicNetworkhistoryArchiveURLs
+		config.SerializedCaptiveCoreToml = ledgerbackend.PubnetDefaultConfig
+
+	case Testnet:
+		networkPassPhrase = network.TestNetworkPassphrase
+		networkArchiveUrls = network.TestNetworkhistoryArchiveURLs
+		config.SerializedCaptiveCoreToml = ledgerbackend.TestnetDefaultConfig
+
+	default:
+		return errors.New("invalid captive core config, " +
+			"preconfigured_network must be set to 'pubnet' or 'testnet' or network_passphrase, history_archive_urls," +
+			" and captive_core_toml_path must be set")
+	}
+
+	if config.StellarCoreConfig.NetworkPassphrase == "" {
+		config.StellarCoreConfig.NetworkPassphrase = networkPassPhrase
+	}
+
+	if len(config.StellarCoreConfig.HistoryArchiveUrls) < 1 {
+		config.StellarCoreConfig.HistoryArchiveUrls = networkArchiveUrls
+	}
+
+	if config.StellarCoreConfig.CaptiveCoreTomlPath != "" {
+		if config.SerializedCaptiveCoreToml, err = os.ReadFile(config.StellarCoreConfig.CaptiveCoreTomlPath); err != nil {
+			return errors.Wrap(err, "Failed to load captive-core-toml-path file")
+		}
+	}
+
 	return nil
 }
 
 func (config *Config) adjustLedgerRange() {
-
 	// Check if either the start or end ledger does not fall on the "LedgersPerFile" boundary
 	// and adjust the start and end ledger accordingly.
 	// Align the start ledger to the nearest "LedgersPerFile" boundary.
-	config.StartLedger = config.LedgerBatchConfig.GetSequenceNumberStartBoundary(config.StartLedger)
+	config.StartLedger = config.DataStoreConfig.Schema.GetSequenceNumberStartBoundary(config.StartLedger)
 
 	// Ensure that the adjusted start ledger is at least 2.
 	config.StartLedger = ordered.Max(2, config.StartLedger)
 
 	// Align the end ledger (for bounded cases) to the nearest "LedgersPerFile" boundary.
 	if config.EndLedger != 0 {
-		config.EndLedger = config.LedgerBatchConfig.GetSequenceNumberEndBoundary(config.EndLedger)
+		config.EndLedger = config.DataStoreConfig.Schema.GetSequenceNumberEndBoundary(config.EndLedger)
 	}
 
 	logger.Infof("Computed effective export boundary ledger range: start=%d, end=%d", config.StartLedger, config.EndLedger)
diff --git a/exp/services/ledgerexporter/internal/exportmanager.go b/exp/services/ledgerexporter/internal/exportmanager.go
index ae85c072..af2633d6 100644
--- a/exp/services/ledgerexporter/internal/exportmanager.go
+++ b/exp/services/ledgerexporter/internal/exportmanager.go
@@ -8,21 +8,29 @@ import (
 	"github.com/prometheus/client_golang/prometheus"
 
 	"github.com/stellar/go/ingest/ledgerbackend"
+	"github.com/stellar/go/support/datastore"
 	"github.com/stellar/go/xdr"
 )
 
 type ExportManager struct {
-	config             *Config
+	dataStoreSchema    datastore.DataStoreSchema
 	ledgerBackend      ledgerbackend.LedgerBackend
 	currentMetaArchive *xdr.LedgerCloseMetaBatch
 	queue              UploadQueue
 	latestLedgerMetric *prometheus.GaugeVec
+	networkPassPhrase  string
+	coreVersion        string
 }
 
 // NewExportManager creates a new ExportManager with the provided configuration.
-func NewExportManager(config *Config, backend ledgerbackend.LedgerBackend, queue UploadQueue, prometheusRegistry *prometheus.Registry) (*ExportManager, error) {
-	if config.LedgerBatchConfig.LedgersPerFile < 1 {
-		return nil, errors.Errorf("Invalid ledgers per file (%d): must be at least 1", config.LedgerBatchConfig.LedgersPerFile)
+func NewExportManager(dataStoreSchema datastore.DataStoreSchema,
+	backend ledgerbackend.LedgerBackend,
+	queue UploadQueue,
+	prometheusRegistry *prometheus.Registry,
+	networkPassPhrase string,
+	coreVersion string) (*ExportManager, error) {
+	if dataStoreSchema.LedgersPerFile < 1 {
+		return nil, errors.Errorf("Invalid ledgers per file (%d): must be at least 1", dataStoreSchema.LedgersPerFile)
 	}
 
 	latestLedgerMetric := prometheus.NewGaugeVec(prometheus.GaugeOpts{
@@ -32,10 +40,12 @@ func NewExportManager(config *Config, backend ledgerbackend.LedgerBackend, queue
 	prometheusRegistry.MustRegister(latestLedgerMetric)
 
 	return &ExportManager{
-		config:             config,
+		dataStoreSchema:    dataStoreSchema,
 		ledgerBackend:      backend,
 		queue:              queue,
 		latestLedgerMetric: latestLedgerMetric,
+		networkPassPhrase:  networkPassPhrase,
+		coreVersion:        coreVersion,
 	}, nil
 }
 
@@ -44,16 +54,16 @@ func (e *ExportManager) AddLedgerCloseMeta(ctx context.Context, ledgerCloseMeta
 	ledgerSeq := ledgerCloseMeta.LedgerSequence()
 
 	// Determine the object key for the given ledger sequence
-	objectKey := e.config.LedgerBatchConfig.GetObjectKeyFromSequenceNumber(ledgerSeq)
+	objectKey := e.dataStoreSchema.GetObjectKeyFromSequenceNumber(ledgerSeq)
 
 	if e.currentMetaArchive == nil {
-		endSeq := ledgerSeq + e.config.LedgerBatchConfig.LedgersPerFile - 1
-		if ledgerSeq < e.config.LedgerBatchConfig.LedgersPerFile {
+		endSeq := ledgerSeq + e.dataStoreSchema.LedgersPerFile - 1
+		if ledgerSeq < e.dataStoreSchema.LedgersPerFile {
 			// Special case: Adjust the end ledger sequence for the first batch.
 			// Since the start ledger is 2 instead of 0, we want to ensure that the end ledger sequence
 			// does not exceed LedgersPerFile.
 			// For example, if LedgersPerFile is 64, the file name for the first batch should be 0-63, not 2-66.
-			endSeq = e.config.LedgerBatchConfig.LedgersPerFile - 1
+			endSeq = e.dataStoreSchema.LedgersPerFile - 1
 		}
 
 		// Create a new LedgerCloseMetaBatch
@@ -65,7 +75,7 @@ func (e *ExportManager) AddLedgerCloseMeta(ctx context.Context, ledgerCloseMeta
 	}
 
 	if ledgerSeq >= uint32(e.currentMetaArchive.EndSequence) {
-		ledgerMetaArchive, err := NewLedgerMetaArchiveFromXDR(e.config, objectKey, *e.currentMetaArchive)
+		ledgerMetaArchive, err := NewLedgerMetaArchiveFromXDR(e.networkPassPhrase, e.coreVersion, objectKey, *e.currentMetaArchive)
 		if err != nil {
 			return err
 		}
diff --git a/exp/services/ledgerexporter/internal/ledger_meta_archive.go b/exp/services/ledgerexporter/internal/ledger_meta_archive.go
index 583f6a3e..f63a77f2 100644
--- a/exp/services/ledgerexporter/internal/ledger_meta_archive.go
+++ b/exp/services/ledgerexporter/internal/ledger_meta_archive.go
@@ -14,7 +14,7 @@ type LedgerMetaArchive struct {
 }
 
 // NewLedgerMetaArchiveFromXDR creates a new LedgerMetaArchive instance.
-func NewLedgerMetaArchiveFromXDR(config *Config, key string, data xdr.LedgerCloseMetaBatch) (*LedgerMetaArchive, error) {
+func NewLedgerMetaArchiveFromXDR(networkPassPhrase string, coreVersion string, key string, data xdr.LedgerCloseMetaBatch) (*LedgerMetaArchive, error) {
 	startLedger, err := data.GetLedger(uint32(data.StartSequence))
 	if err != nil {
 		return &LedgerMetaArchive{}, err
@@ -33,10 +33,10 @@ func NewLedgerMetaArchiveFromXDR(config *Config, key string, data xdr.LedgerClos
 			EndLedger:            endLedger.LedgerSequence(),
 			StartLedgerCloseTime: startLedger.LedgerCloseTime(),
 			EndLedgerCloseTime:   endLedger.LedgerCloseTime(),
-			Network:              config.Network,
+			NetworkPassPhrase:    networkPassPhrase,
 			CompressionType:      compressxdr.DefaultCompressor.Name(),
 			ProtocolVersion:      endLedger.ProtocolVersion(),
-			CoreVersion:          config.CoreVersion,
+			CoreVersion:          coreVersion,
 			Version:              version,
 		},
 	}, nil
diff --git a/exp/services/ledgerexporter/internal/main.go b/exp/services/ledgerexporter/internal/main.go
new file mode 100644
index 00000000..d1409eb8
--- /dev/null
+++ b/exp/services/ledgerexporter/internal/main.go
@@ -0,0 +1,94 @@
+package ledgerexporter
+
+import (
+	"fmt"
+
+	"github.com/spf13/cobra"
+	"github.com/spf13/pflag"
+	"github.com/spf13/viper"
+	"github.com/stellar/go/support/strutils"
+)
+
+var (
+	ledgerExporterCmdRunner = func(runtimeSettings RuntimeSettings) error {
+		app := NewApp()
+		return app.Run(runtimeSettings)
+	}
+	rootCmd, scanAndFillCmd, appendCmd *cobra.Command
+)
+
+func Execute() error {
+	defineCommands()
+	return rootCmd.Execute()
+}
+
+func defineCommands() {
+	rootCmd = &cobra.Command{
+		Use:   "ledgerexporter",
+		Short: "Export Stellar network ledger data to a remote data store",
+		Long:  "Converts ledger meta data from Stellar network into static data and exports it remote data storage.",
+
+		RunE: func(cmd *cobra.Command, args []string) error {
+			return fmt.Errorf("please specify one of the availble sub-commands to initiate export")
+		},
+	}
+	scanAndFillCmd = &cobra.Command{
+		Use:   "scan-and-fill",
+		Short: "scans the entire bounded requested range between 'start' and 'end' flags and exports only the ledgers which are missing from the data lake.",
+		Long:  "scans the entire bounded requested range between 'start' and 'end' flags and exports only the ledgers which are missing from the data lake.",
+		RunE: func(cmd *cobra.Command, args []string) error {
+			settings := bindCliParameters(cmd.PersistentFlags().Lookup("start"),
+				cmd.PersistentFlags().Lookup("end"),
+				cmd.PersistentFlags().Lookup("config-file"),
+			)
+			settings.Mode = ScanFill
+			return ledgerExporterCmdRunner(settings)
+		},
+	}
+	appendCmd = &cobra.Command{
+		Use:   "append",
+		Short: "export ledgers beginning with the first missing ledger after the specified 'start' ledger and resumes exporting from there",
+		Long:  "export ledgers beginning with the first missing ledger after the specified 'start' ledger and resumes exporting from there",
+		RunE: func(cmd *cobra.Command, args []string) error {
+			settings := bindCliParameters(cmd.PersistentFlags().Lookup("start"),
+				cmd.PersistentFlags().Lookup("end"),
+				cmd.PersistentFlags().Lookup("config-file"),
+			)
+			settings.Mode = Append
+			return ledgerExporterCmdRunner(settings)
+		},
+	}
+
+	rootCmd.AddCommand(scanAndFillCmd)
+	rootCmd.AddCommand(appendCmd)
+
+	scanAndFillCmd.PersistentFlags().Uint32P("start", "s", 0, "Starting ledger (inclusive), must be set to a value greater than 1")
+	scanAndFillCmd.PersistentFlags().Uint32P("end", "e", 0, "Ending ledger (inclusive), must be set to value greater than 'start' and less than the network's current ledger")
+	scanAndFillCmd.PersistentFlags().String("config-file", "config.toml", "Path to the TOML config file. Defaults to 'config.toml' on runtime working directory path.")
+	viper.BindPFlags(scanAndFillCmd.PersistentFlags())
+
+	appendCmd.PersistentFlags().Uint32P("start", "s", 0, "Starting ledger (inclusive), must be set to a value greater than 1")
+	appendCmd.PersistentFlags().Uint32P("end", "e", 0, "Ending ledger (inclusive), optional, setting to non-zero means bounded mode, "+
+		"only export ledgers from 'start' up to 'end' value which must be greater than 'start' and less than the network's current ledger. "+
+		"If 'end' is absent or '0' means unbounded mode, exporter will continue to run indefintely and export the latest closed ledgers from network as they are generated in real time.")
+	appendCmd.PersistentFlags().String("config-file", "config.toml", "Path to the TOML config file. Defaults to 'config.toml' on runtime working directory path.")
+	viper.BindPFlags(appendCmd.PersistentFlags())
+}
+
+func bindCliParameters(startFlag *pflag.Flag, endFlag *pflag.Flag, configFileFlag *pflag.Flag) RuntimeSettings {
+	settings := RuntimeSettings{}
+
+	viper.BindPFlag(startFlag.Name, startFlag)
+	viper.BindEnv(startFlag.Name, strutils.KebabToConstantCase(startFlag.Name))
+	settings.StartLedger = viper.GetUint32(startFlag.Name)
+
+	viper.BindPFlag(endFlag.Name, endFlag)
+	viper.BindEnv(endFlag.Name, strutils.KebabToConstantCase(endFlag.Name))
+	settings.EndLedger = viper.GetUint32(endFlag.Name)
+
+	viper.BindPFlag(configFileFlag.Name, configFileFlag)
+	viper.BindEnv(configFileFlag.Name, strutils.KebabToConstantCase(configFileFlag.Name))
+	settings.ConfigFilePath = viper.GetString(configFileFlag.Name)
+
+	return settings
+}
diff --git a/exp/services/ledgerexporter/main.go b/exp/services/ledgerexporter/main.go
index 63e09498..6d38c69d 100644
--- a/exp/services/ledgerexporter/main.go
+++ b/exp/services/ledgerexporter/main.go
@@ -1,25 +1,16 @@
 package main
 
 import (
-	"flag"
+	"fmt"
+	"os"
 
 	exporter "github.com/stellar/go/exp/services/ledgerexporter/internal"
 )
 
 func main() {
-	flags := exporter.Flags{}
-	startLedger := uint(0)
-	endLedger := uint(0)
-	flag.UintVar(&startLedger, "start", 0, "Starting ledger")
-	flag.UintVar(&endLedger, "end", 0, "Ending ledger (inclusive)")
-	flag.StringVar(&flags.ConfigFilePath, "config-file", "config.toml", "Path to the TOML config file")
-	flag.BoolVar(&flags.Resume, "resume", false, "Attempt to find a resumable starting point on remote data store")
-	flag.UintVar(&flags.AdminPort, "admin-port", 0, "Admin HTTP port for prometheus metrics")
-
-	flag.Parse()
-	flags.StartLedger = uint32(startLedger)
-	flags.EndLedger = uint32(endLedger)
-
-	app := exporter.NewApp(flags)
-	app.Run()
+	err := exporter.Execute()
+	if err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
 }
diff --git a/ingest/ledgerbackend/buffered_storage_backend.go b/ingest/ledgerbackend/buffered_storage_backend.go
index 41ff4a94..4a353bfe 100644
--- a/ingest/ledgerbackend/buffered_storage_backend.go
+++ b/ingest/ledgerbackend/buffered_storage_backend.go
@@ -18,7 +18,7 @@ import (
 var _ LedgerBackend = (*BufferedStorageBackend)(nil)
 
 type BufferedStorageBackendConfig struct {
-	LedgerBatchConfig datastore.LedgerBatchConfig
+	LedgerBatchConfig datastore.DataStoreSchema
 	DataStore         datastore.DataStore
 	BufferSize        uint32
 	NumWorkers        uint32
diff --git a/support/datastore/datastore.go b/support/datastore/datastore.go
index 0a92857b..e7e99934 100644
--- a/support/datastore/datastore.go
+++ b/support/datastore/datastore.go
@@ -10,6 +10,7 @@ import (
 type DataStoreConfig struct {
 	Type   string            `toml:"type"`
 	Params map[string]string `toml:"params"`
+	Schema DataStoreSchema   `toml:"schema"`
 }
 
 // DataStore defines an interface for interacting with data storage
@@ -24,14 +25,14 @@ type DataStore interface {
 }
 
 // NewDataStore factory, it creates a new DataStore based on the config type
-func NewDataStore(ctx context.Context, datastoreConfig DataStoreConfig, network string) (DataStore, error) {
+func NewDataStore(ctx context.Context, datastoreConfig DataStoreConfig) (DataStore, error) {
 	switch datastoreConfig.Type {
 	case "GCS":
 		destinationBucketPath, ok := datastoreConfig.Params["destination_bucket_path"]
 		if !ok {
 			return nil, errors.Errorf("Invalid GCS config, no destination_bucket_path")
 		}
-		return NewGCSDataStore(ctx, destinationBucketPath, network)
+		return NewGCSDataStore(ctx, destinationBucketPath)
 	default:
 		return nil, errors.Errorf("Invalid datastore type %v, not supported", datastoreConfig.Type)
 	}
diff --git a/support/datastore/gcs_datastore.go b/support/datastore/gcs_datastore.go
index c6b22b0f..cdedea08 100644
--- a/support/datastore/gcs_datastore.go
+++ b/support/datastore/gcs_datastore.go
@@ -26,19 +26,19 @@ type GCSDataStore struct {
 	prefix string
 }
 
-func NewGCSDataStore(ctx context.Context, bucketPath string, network string) (DataStore, error) {
+func NewGCSDataStore(ctx context.Context, bucketPath string) (DataStore, error) {
 	client, err := storage.NewClient(ctx)
 	if err != nil {
 		return nil, err
 	}
 
-	return FromGCSClient(ctx, client, bucketPath, network)
+	return FromGCSClient(ctx, client, bucketPath)
 }
 
-func FromGCSClient(ctx context.Context, client *storage.Client, bucketPath string, network string) (DataStore, error) {
+func FromGCSClient(ctx context.Context, client *storage.Client, bucketPath string) (DataStore, error) {
 	// append the gcs:// scheme to enable usage of the url package reliably to
 	// get parse bucket name which is first path segment as URL.Host
-	gcsBucketURL := fmt.Sprintf("gcs://%s/%s", bucketPath, network)
+	gcsBucketURL := fmt.Sprintf("gcs://%s", bucketPath)
 	parsed, err := url.Parse(gcsBucketURL)
 	if err != nil {
 		return nil, err
diff --git a/support/datastore/history_archive.go b/support/datastore/history_archive.go
index bc692fb7..123240dc 100644
--- a/support/datastore/history_archive.go
+++ b/support/datastore/history_archive.go
@@ -10,10 +10,12 @@ import (
 	"github.com/stellar/go/support/storage"
 )
 
-const Pubnet = "pubnet"
-const Testnet = "testnet"
+const (
+	Pubnet  = "pubnet"
+	Testnet = "testnet"
+)
 
-func CreateHistoryArchiveFromNetworkName(ctx context.Context, networkName string) (historyarchive.ArchiveInterface, error) {
+func CreateHistoryArchiveFromNetworkName(ctx context.Context, networkName string, userAgent string) (historyarchive.ArchiveInterface, error) {
 	var historyArchiveUrls []string
 	switch networkName {
 	case Pubnet:
@@ -26,7 +28,7 @@ func CreateHistoryArchiveFromNetworkName(ctx context.Context, networkName string
 
 	return historyarchive.NewArchivePool(historyArchiveUrls, historyarchive.ArchiveOptions{
 		ConnectOptions: storage.ConnectOptions{
-			UserAgent: "ledger-exporter",
+			UserAgent: userAgent,
 			Context:   ctx,
 		},
 	})
diff --git a/support/datastore/ledgerbatch_config.go b/support/datastore/ledgerbatch_config.go
index 4ebac0d1..e70bfb5f 100644
--- a/support/datastore/ledgerbatch_config.go
+++ b/support/datastore/ledgerbatch_config.go
@@ -7,24 +7,24 @@ import (
 	"github.com/stellar/go/support/compressxdr"
 )
 
-type LedgerBatchConfig struct {
+type DataStoreSchema struct {
 	LedgersPerFile    uint32 `toml:"ledgers_per_file"`
 	FilesPerPartition uint32 `toml:"files_per_partition"`
 }
 
-func (ec LedgerBatchConfig) GetSequenceNumberStartBoundary(ledgerSeq uint32) uint32 {
+func (ec DataStoreSchema) GetSequenceNumberStartBoundary(ledgerSeq uint32) uint32 {
 	if ec.LedgersPerFile == 0 {
 		return 0
 	}
 	return (ledgerSeq / ec.LedgersPerFile) * ec.LedgersPerFile
 }
 
-func (ec LedgerBatchConfig) GetSequenceNumberEndBoundary(ledgerSeq uint32) uint32 {
+func (ec DataStoreSchema) GetSequenceNumberEndBoundary(ledgerSeq uint32) uint32 {
 	return ec.GetSequenceNumberStartBoundary(ledgerSeq) + ec.LedgersPerFile - 1
 }
 
 // GetObjectKeyFromSequenceNumber generates the object key name from the ledger sequence number based on configuration.
-func (ec LedgerBatchConfig) GetObjectKeyFromSequenceNumber(ledgerSeq uint32) string {
+func (ec DataStoreSchema) GetObjectKeyFromSequenceNumber(ledgerSeq uint32) string {
 	var objectKey string
 
 	if ec.FilesPerPartition > 1 {
diff --git a/support/datastore/metadata.go b/support/datastore/metadata.go
index db734bbd..bd943b19 100644
--- a/support/datastore/metadata.go
+++ b/support/datastore/metadata.go
@@ -9,7 +9,7 @@ type MetaData struct {
 	EndLedgerCloseTime   int64
 	ProtocolVersion      uint32
 	CoreVersion          string
-	Network              string
+	NetworkPassPhrase    string
 	CompressionType      string
 	Version              string
 }
@@ -22,7 +22,7 @@ func (m MetaData) ToMap() map[string]string {
 		"end-ledger-close-time":   strconv.FormatInt(m.EndLedgerCloseTime, 10),
 		"protocol-version":        strconv.FormatInt(int64(m.ProtocolVersion), 10),
 		"core-version":            m.CoreVersion,
-		"network":                 m.Network,
+		"network-passphrase":      m.NetworkPassPhrase,
 		"compression-type":        m.CompressionType,
 		"version":                 m.Version,
 	}
@@ -71,7 +71,7 @@ func NewMetaDataFromMap(data map[string]string) (MetaData, error) {
 	}
 
 	metaData.CoreVersion = data["core-version"]
-	metaData.Network = data["network"]
+	metaData.NetworkPassPhrase = data["network-passphrase"]
 	metaData.CompressionType = data["compression-type"]
 	metaData.Version = data["version"]
 
diff --git a/support/datastore/resumeablemanager.go b/support/datastore/resumeablemanager.go
index f552dcf1..35031d73 100644
--- a/support/datastore/resumeablemanager.go
+++ b/support/datastore/resumeablemanager.go
@@ -37,19 +37,16 @@ type ResumableManager interface {
 }
 
 type resumableManagerService struct {
-	network           string
-	ledgerBatchConfig LedgerBatchConfig
+	ledgerBatchConfig DataStoreSchema
 	dataStore         DataStore
 	archive           historyarchive.ArchiveInterface
 }
 
 func NewResumableManager(dataStore DataStore,
-	network string,
-	ledgerBatchConfig LedgerBatchConfig,
+	ledgerBatchConfig DataStoreSchema,
 	archive historyarchive.ArchiveInterface) ResumableManager {
 	return &resumableManagerService{
 		ledgerBatchConfig: ledgerBatchConfig,
-		network:           network,
 		dataStore:         dataStore,
 		archive:           archive,
 	}
@@ -60,7 +57,7 @@ func (rm resumableManagerService) FindStart(ctx context.Context, start, end uint
 		return 0, false, errors.New("Invalid start value, must be greater than zero")
 	}
 
-	log.WithField("start", start).WithField("end", end).WithField("network", rm.network)
+	log.WithField("start", start).WithField("end", end)
 
 	networkLatest := uint32(0)
 	if end < 1 {
@@ -71,7 +68,7 @@ func (rm resumableManagerService) FindStart(ctx context.Context, start, end uint
 			return 0, false, err
 		}
 		networkLatest = networkLatest + (GetHistoryArchivesCheckPointFrequency() * 2)
-		log.Infof("Resumability computed effective latest network ledger including padding of checkpoint frequency to be %d + for network=%v", networkLatest, rm.network)
+		log.Infof("Resumability computed effective latest network ledger including padding of checkpoint frequency to be %d", networkLatest)
 
 		if start > networkLatest {
 			// requested to start at a point beyond the latest network, resume not applicable.


Test patch:
diff --git a/exp/services/ledgerexporter/internal/app_test.go b/exp/services/ledgerexporter/internal/app_test.go
index 2063fd21..bb715baa 100644
--- a/exp/services/ledgerexporter/internal/app_test.go
+++ b/exp/services/ledgerexporter/internal/app_test.go
@@ -12,7 +12,7 @@ import (
 func TestApplyResumeHasStartError(t *testing.T) {
 	ctx := context.Background()
 	app := &App{}
-	app.config = &Config{StartLedger: 10, EndLedger: 19, Resume: true}
+	app.config = &Config{StartLedger: 10, EndLedger: 19, Mode: Append}
 	mockResumableManager := &datastore.MockResumableManager{}
 	mockResumableManager.On("FindStart", ctx, uint32(10), uint32(19)).Return(uint32(0), false, errors.New("start error")).Once()
 
@@ -24,7 +24,7 @@ func TestApplyResumeHasStartError(t *testing.T) {
 func TestApplyResumeDatastoreComplete(t *testing.T) {
 	ctx := context.Background()
 	app := &App{}
-	app.config = &Config{StartLedger: 10, EndLedger: 19, Resume: true}
+	app.config = &Config{StartLedger: 10, EndLedger: 19, Mode: Append}
 	mockResumableManager := &datastore.MockResumableManager{}
 	mockResumableManager.On("FindStart", ctx, uint32(10), uint32(19)).Return(uint32(0), false, nil).Once()
 
@@ -38,10 +38,10 @@ func TestApplyResumeInvalidDataStoreLedgersPerFileBoundary(t *testing.T) {
 	ctx := context.Background()
 	app := &App{}
 	app.config = &Config{
-		StartLedger:       3,
-		EndLedger:         9,
-		Resume:            true,
-		LedgerBatchConfig: datastore.LedgerBatchConfig{LedgersPerFile: 10, FilesPerPartition: 50},
+		StartLedger:     3,
+		EndLedger:       9,
+		Mode:            Append,
+		DataStoreConfig: datastore.DataStoreConfig{Schema: datastore.DataStoreSchema{LedgersPerFile: 10, FilesPerPartition: 50}},
 	}
 	mockResumableManager := &datastore.MockResumableManager{}
 	// simulate the datastore has inconsistent data,
@@ -58,10 +58,10 @@ func TestApplyResumeWithPartialRemoteDataPresent(t *testing.T) {
 	ctx := context.Background()
 	app := &App{}
 	app.config = &Config{
-		StartLedger:       10,
-		EndLedger:         99,
-		Resume:            true,
-		LedgerBatchConfig: datastore.LedgerBatchConfig{LedgersPerFile: 10, FilesPerPartition: 50},
+		StartLedger:     10,
+		EndLedger:       99,
+		Mode:            Append,
+		DataStoreConfig: datastore.DataStoreConfig{Schema: datastore.DataStoreSchema{LedgersPerFile: 10, FilesPerPartition: 50}},
 	}
 	mockResumableManager := &datastore.MockResumableManager{}
 	// simulates a data store that had ledger files populated up to seq=49, so the first absent ledger would be 50
@@ -77,10 +77,10 @@ func TestApplyResumeWithNoRemoteDataPresent(t *testing.T) {
 	ctx := context.Background()
 	app := &App{}
 	app.config = &Config{
-		StartLedger:       10,
-		EndLedger:         99,
-		Resume:            true,
-		LedgerBatchConfig: datastore.LedgerBatchConfig{LedgersPerFile: 10, FilesPerPartition: 50},
+		StartLedger:     10,
+		EndLedger:       99,
+		Mode:            Append,
+		DataStoreConfig: datastore.DataStoreConfig{Schema: datastore.DataStoreSchema{LedgersPerFile: 10, FilesPerPartition: 50}},
 	}
 	mockResumableManager := &datastore.MockResumableManager{}
 	// simulates a data store that had no data in the requested range
@@ -99,10 +99,10 @@ func TestApplyResumeWithNoRemoteDataAndRequestFromGenesis(t *testing.T) {
 	ctx := context.Background()
 	app := &App{}
 	app.config = &Config{
-		StartLedger:       2,
-		EndLedger:         99,
-		Resume:            true,
-		LedgerBatchConfig: datastore.LedgerBatchConfig{LedgersPerFile: 10, FilesPerPartition: 50},
+		StartLedger:     2,
+		EndLedger:       99,
+		Mode:            Append,
+		DataStoreConfig: datastore.DataStoreConfig{Schema: datastore.DataStoreSchema{LedgersPerFile: 10, FilesPerPartition: 50}},
 	}
 	mockResumableManager := &datastore.MockResumableManager{}
 	// simulates a data store that had no data in the requested range
diff --git a/exp/services/ledgerexporter/internal/config_test.go b/exp/services/ledgerexporter/internal/config_test.go
index a4ce76cf..37dc5740 100644
--- a/exp/services/ledgerexporter/internal/config_test.go
+++ b/exp/services/ledgerexporter/internal/config_test.go
@@ -7,118 +7,304 @@ import (
 	"os/exec"
 	"testing"
 
+	"github.com/stellar/go/network"
+	"github.com/stellar/go/support/datastore"
+
 	"github.com/stretchr/testify/require"
 
 	"github.com/stellar/go/historyarchive"
 	"github.com/stellar/go/support/errors"
 )
 
-func TestNewConfigResumeEnabled(t *testing.T) {
+func TestNewConfig(t *testing.T) {
 	ctx := context.Background()
 
 	mockArchive := &historyarchive.MockArchive{}
 	mockArchive.On("GetRootHAS").Return(historyarchive.HistoryArchiveState{CurrentLedger: 5}, nil).Once()
 
-	config, err := NewConfig(Flags{StartLedger: 1, EndLedger: 2, ConfigFilePath: "test/test.toml", Resume: true})
-	config.ValidateAndSetLedgerRange(ctx, mockArchive)
+	config, err := NewConfig(
+		RuntimeSettings{StartLedger: 2, EndLedger: 3, ConfigFilePath: "test/test.toml", Mode: Append})
+
+	require.NoError(t, err)
+	err = config.ValidateAndSetLedgerRange(ctx, mockArchive)
 	require.NoError(t, err)
+	require.Equal(t, config.StellarCoreConfig.Network, "pubnet")
 	require.Equal(t, config.DataStoreConfig.Type, "ABC")
-	require.Equal(t, config.LedgerBatchConfig.FilesPerPartition, uint32(1))
-	require.Equal(t, config.LedgerBatchConfig.LedgersPerFile, uint32(3))
-	require.True(t, config.Resume)
+	require.Equal(t, config.DataStoreConfig.Schema.FilesPerPartition, uint32(1))
+	require.Equal(t, config.DataStoreConfig.Schema.LedgersPerFile, uint32(3))
+	require.Equal(t, config.UserAgent, "ledgerexporter")
+	require.True(t, config.Resumable())
 	url, ok := config.DataStoreConfig.Params["destination_bucket_path"]
 	require.True(t, ok)
-	require.Equal(t, url, "your-bucket-name/subpath")
+	require.Equal(t, url, "your-bucket-name/subpath/testnet")
+	mockArchive.AssertExpectations(t)
 }
 
-func TestNewConfigResumeDisabled(t *testing.T) {
+func TestGenerateHistoryArchiveFromPreconfiguredNetwork(t *testing.T) {
+	ctx := context.Background()
+	config, err := NewConfig(
+		RuntimeSettings{StartLedger: 2, EndLedger: 3, ConfigFilePath: "test/valid_captive_core_preconfigured.toml", Mode: Append})
+	require.NoError(t, err)
 
-	mockArchive := &historyarchive.MockArchive{}
-	mockArchive.On("GetRootHAS").Return(historyarchive.HistoryArchiveState{CurrentLedger: 5}, nil).Once()
+	_, err = config.GenerateHistoryArchive(ctx)
+	require.NoError(t, err)
+}
+
+func TestGenerateHistoryArchiveFromManulConfiguredNetwork(t *testing.T) {
+	ctx := context.Background()
+	config, err := NewConfig(
+		RuntimeSettings{StartLedger: 2, EndLedger: 3, ConfigFilePath: "test/valid_captive_core_manual.toml", Mode: Append})
+	require.NoError(t, err)
+
+	_, err = config.GenerateHistoryArchive(ctx)
+	require.NoError(t, err)
+}
+
+func TestNewConfigUserAgent(t *testing.T) {
+	config, err := NewConfig(
+		RuntimeSettings{StartLedger: 2, EndLedger: 3, ConfigFilePath: "test/useragent.toml"})
+	require.NoError(t, err)
+	require.Equal(t, config.UserAgent, "useragent_x")
+}
+
+func TestResumeDisabled(t *testing.T) {
+	// resumable is only enabled when mode is Append
+	config, err := NewConfig(
+		RuntimeSettings{StartLedger: 2, EndLedger: 3, ConfigFilePath: "test/test.toml", Mode: ScanFill})
+	require.NoError(t, err)
+	require.False(t, config.Resumable())
+}
+
+func TestInvalidConfigFilePath(t *testing.T) {
+	_, err := NewConfig(
+		RuntimeSettings{ConfigFilePath: "test/notfound.toml"})
+	require.ErrorContains(t, err, "config file test/notfound.toml was not found")
+}
+
+func TestNoCaptiveCoreBin(t *testing.T) {
+	cfg, err := NewConfig(
+		RuntimeSettings{ConfigFilePath: "test/no_core_bin.toml"})
+	require.NoError(t, err)
+
+	_, err = cfg.GenerateCaptiveCoreConfig("")
+	require.ErrorContains(t, err, "Invalid captive core config, no stellar-core binary path was provided.")
+}
+
+func TestDefaultCaptiveCoreBin(t *testing.T) {
+	cfg, err := NewConfig(
+		RuntimeSettings{ConfigFilePath: "test/no_core_bin.toml"})
+	require.NoError(t, err)
+
+	cmdOut = "v20.2.0-2-g6e73c0a88\n"
+	ccConfig, err := cfg.GenerateCaptiveCoreConfig("/test/default/stellar-core")
+	require.NoError(t, err)
+	require.Equal(t, ccConfig.BinaryPath, "/test/default/stellar-core")
+}
+
+func TestInvalidCaptiveCorePreconfiguredNetwork(t *testing.T) {
+	_, err := NewConfig(
+		RuntimeSettings{ConfigFilePath: "test/invalid_preconfigured_network.toml"})
+
+	require.ErrorContains(t, err, "invalid captive core config")
+}
+
+func TestValidCaptiveCorePreconfiguredNetwork(t *testing.T) {
+	cfg, err := NewConfig(
+		RuntimeSettings{ConfigFilePath: "test/valid_captive_core_preconfigured.toml"})
+	require.NoError(t, err)
+
+	require.Equal(t, cfg.StellarCoreConfig.NetworkPassphrase, network.PublicNetworkPassphrase)
+	require.Equal(t, cfg.StellarCoreConfig.HistoryArchiveUrls, network.PublicNetworkhistoryArchiveURLs)
+
+	cmdOut = "v20.2.0-2-g6e73c0a88\n"
+	ccConfig, err := cfg.GenerateCaptiveCoreConfig("")
+	require.NoError(t, err)
+
+	// validates that ingest/ledgerbackend/configs/captive-core-pubnet.cfg was loaded
+	require.Equal(t, ccConfig.BinaryPath, "test/stellar-core")
+	require.Equal(t, ccConfig.NetworkPassphrase, network.PublicNetworkPassphrase)
+	require.Equal(t, ccConfig.HistoryArchiveURLs, network.PublicNetworkhistoryArchiveURLs)
+	require.Empty(t, ccConfig.Toml.HistoryEntries)
+	require.Len(t, ccConfig.Toml.Validators, 23)
+	require.Equal(t, ccConfig.Toml.Validators[0].Name, "Boötes")
+}
 
-	// resume disabled by default
-	config, err := NewConfig(Flags{StartLedger: 1, EndLedger: 2, ConfigFilePath: "test/test.toml"})
+func TestValidCaptiveCoreManualNetwork(t *testing.T) {
+	cfg, err := NewConfig(
+		RuntimeSettings{ConfigFilePath: "test/valid_captive_core_manual.toml"})
 	require.NoError(t, err)
-	require.False(t, config.Resume)
+	require.Equal(t, cfg.CoreVersion, "")
+	require.Equal(t, cfg.StellarCoreConfig.NetworkPassphrase, "test")
+	require.Equal(t, cfg.StellarCoreConfig.HistoryArchiveUrls, []string{"http://testarchive"})
+
+	cmdOut = "v20.2.0-2-g6e73c0a88\n"
+	ccConfig, err := cfg.GenerateCaptiveCoreConfig("")
+	require.NoError(t, err)
+
+	require.Equal(t, ccConfig.BinaryPath, "test/stellar-core")
+	require.Equal(t, ccConfig.NetworkPassphrase, "test")
+	require.Equal(t, ccConfig.HistoryArchiveURLs, []string{"http://testarchive"})
+	require.Empty(t, ccConfig.Toml.HistoryEntries)
+	require.Len(t, ccConfig.Toml.Validators, 1)
+	require.Equal(t, ccConfig.Toml.Validators[0].Name, "local_core")
+	require.Equal(t, cfg.CoreVersion, "v20.2.0-2-g6e73c0a88")
 }
 
-func TestInvalidTomlConfig(t *testing.T) {
+func TestValidCaptiveCoreOverridenToml(t *testing.T) {
+	cfg, err := NewConfig(
+		RuntimeSettings{ConfigFilePath: "test/valid_captive_core_override.toml"})
+	require.NoError(t, err)
+	require.Equal(t, cfg.StellarCoreConfig.NetworkPassphrase, network.PublicNetworkPassphrase)
+	require.Equal(t, cfg.StellarCoreConfig.HistoryArchiveUrls, network.PublicNetworkhistoryArchiveURLs)
+
+	cmdOut = "v20.2.0-2-g6e73c0a88\n"
+	ccConfig, err := cfg.GenerateCaptiveCoreConfig("")
+	require.NoError(t, err)
 
-	_, err := NewConfig(Flags{StartLedger: 1, EndLedger: 2, ConfigFilePath: "test/no_network.toml", Resume: true})
-	require.ErrorContains(t, err, "Invalid TOML config")
+	// the external core cfg file should have applied over the preconf'd network config
+	require.Equal(t, ccConfig.BinaryPath, "test/stellar-core")
+	require.Equal(t, ccConfig.NetworkPassphrase, network.PublicNetworkPassphrase)
+	require.Equal(t, ccConfig.HistoryArchiveURLs, network.PublicNetworkhistoryArchiveURLs)
+	require.Empty(t, ccConfig.Toml.HistoryEntries)
+	require.Len(t, ccConfig.Toml.Validators, 1)
+	require.Equal(t, ccConfig.Toml.Validators[0].Name, "local_core")
+	require.Equal(t, cfg.CoreVersion, "v20.2.0-2-g6e73c0a88")
+}
+
+func TestValidCaptiveCoreOverridenArchiveUrls(t *testing.T) {
+	cfg, err := NewConfig(
+		RuntimeSettings{ConfigFilePath: "test/valid_captive_core_override_archives.toml"})
+	require.NoError(t, err)
+
+	require.Equal(t, cfg.StellarCoreConfig.NetworkPassphrase, network.PublicNetworkPassphrase)
+	require.Equal(t, cfg.StellarCoreConfig.HistoryArchiveUrls, []string{"http://testarchive"})
+
+	cmdOut = "v20.2.0-2-g6e73c0a88\n"
+	ccConfig, err := cfg.GenerateCaptiveCoreConfig("")
+	require.NoError(t, err)
+
+	// validates that ingest/ledgerbackend/configs/captive-core-pubnet.cfg was loaded
+	require.Equal(t, ccConfig.BinaryPath, "test/stellar-core")
+	require.Equal(t, ccConfig.NetworkPassphrase, network.PublicNetworkPassphrase)
+	require.Equal(t, ccConfig.HistoryArchiveURLs, []string{"http://testarchive"})
+	require.Empty(t, ccConfig.Toml.HistoryEntries)
+	require.Len(t, ccConfig.Toml.Validators, 23)
+	require.Equal(t, ccConfig.Toml.Validators[0].Name, "Boötes")
+}
+
+func TestInvalidCaptiveCoreTomlPath(t *testing.T) {
+	_, err := NewConfig(
+		RuntimeSettings{ConfigFilePath: "test/invalid_captive_core_toml_path.toml"})
+	require.ErrorContains(t, err, "Failed to load captive-core-toml-path file")
 }
 
 func TestValidateStartAndEndLedger(t *testing.T) {
-	const latestNetworkLedger = uint32(20000)
+	latestNetworkLedger := uint32(20000)
+	latestNetworkLedgerPadding := datastore.GetHistoryArchivesCheckPointFrequency() * 2
 
 	tests := []struct {
 		name        string
 		startLedger uint32
 		endLedger   uint32
 		errMsg      string
+		mode        Mode
+		mockHas     bool
 	}{
 		{
 			name:        "End ledger same as latest ledger",
 			startLedger: 512,
 			endLedger:   512,
-			errMsg:      "",
+			mode:        ScanFill,
+			errMsg:      "invalid end value, must be greater than start",
+			mockHas:     false,
 		},
 		{
 			name:        "End ledger greater than start ledger",
 			startLedger: 512,
 			endLedger:   600,
+			mode:        ScanFill,
 			errMsg:      "",
+			mockHas:     true,
 		},
 		{
-			name:        "No end ledger provided, unbounded mode",
+			name:        "No end ledger provided, append mode, no error",
 			startLedger: 512,
 			endLedger:   0,
+			mode:        Append,
 			errMsg:      "",
+			mockHas:     true,
+		},
+		{
+			name:        "No end ledger provided, scan-and-fill error",
+			startLedger: 512,
+			endLedger:   0,
+			mode:        ScanFill,
+			errMsg:      "invalid end value, unbounded mode not supported, end must be greater than start.",
 		},
 		{
 			name:        "End ledger before start ledger",
 			startLedger: 512,
 			endLedger:   2,
-			errMsg:      "invalid --end value, must be >= --start",
+			mode:        ScanFill,
+			errMsg:      "invalid end value, must be greater than start",
 		},
 		{
 			name:        "End ledger exceeds latest ledger",
 			startLedger: 512,
-			endLedger:   latestNetworkLedger + 1,
-			errMsg: fmt.Sprintf("--end %d exceeds latest network ledger %d",
-				latestNetworkLedger+1, latestNetworkLedger),
+			endLedger:   latestNetworkLedger + latestNetworkLedgerPadding + 1,
+			mode:        ScanFill,
+			mockHas:     true,
+			errMsg: fmt.Sprintf("end %d exceeds latest network ledger %d",
+				latestNetworkLedger+latestNetworkLedgerPadding+1, latestNetworkLedger+latestNetworkLedgerPadding),
 		},
 		{
 			name:        "Start ledger 0",
 			startLedger: 0,
 			endLedger:   2,
-			errMsg:      "",
+			mode:        ScanFill,
+			errMsg:      "invalid start value, must be greater than one.",
+		},
+		{
+			name:        "Start ledger 1",
+			startLedger: 1,
+			endLedger:   2,
+			mode:        ScanFill,
+			errMsg:      "invalid start value, must be greater than one.",
 		},
 		{
 			name:        "Start ledger exceeds latest ledger",
-			startLedger: latestNetworkLedger + 1,
-			endLedger:   0,
-			errMsg: fmt.Sprintf("--start %d exceeds latest network ledger %d",
-				latestNetworkLedger+1, latestNetworkLedger),
+			startLedger: latestNetworkLedger + latestNetworkLedgerPadding + 1,
+			endLedger:   latestNetworkLedger + latestNetworkLedgerPadding + 2,
+			mode:        ScanFill,
+			mockHas:     true,
+			errMsg: fmt.Sprintf("start %d exceeds latest network ledger %d",
+				latestNetworkLedger+latestNetworkLedgerPadding+1, latestNetworkLedger+latestNetworkLedgerPadding),
 		},
 	}
 
 	ctx := context.Background()
 	mockArchive := &historyarchive.MockArchive{}
-	mockArchive.On("GetRootHAS").Return(historyarchive.HistoryArchiveState{CurrentLedger: latestNetworkLedger}, nil).Times(len(tests))
+	mockArchive.On("GetRootHAS").Return(historyarchive.HistoryArchiveState{CurrentLedger: latestNetworkLedger}, nil)
 
+	mockedHasCtr := 0
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			config, err := NewConfig(Flags{StartLedger: tt.startLedger, EndLedger: tt.endLedger, ConfigFilePath: "test/validate_start_end.toml"})
+			if tt.mockHas {
+				mockedHasCtr++
+			}
+			config, err := NewConfig(
+				RuntimeSettings{StartLedger: tt.startLedger, EndLedger: tt.endLedger, ConfigFilePath: "test/validate_start_end.toml", Mode: tt.mode})
 			require.NoError(t, err)
 			err = config.ValidateAndSetLedgerRange(ctx, mockArchive)
 			if tt.errMsg != "" {
+				require.Error(t, err)
 				require.Equal(t, tt.errMsg, err.Error())
 			} else {
 				require.NoError(t, err)
 			}
 		})
 	}
+	mockArchive.AssertNumberOfCalls(t, "GetRootHAS", mockedHasCtr)
 }
 
 func TestAdjustedLedgerRangeBoundedMode(t *testing.T) {
@@ -130,27 +316,19 @@ func TestAdjustedLedgerRangeBoundedMode(t *testing.T) {
 		expectedStart uint32
 		expectedEnd   uint32
 	}{
-		{
-			name:          "Min start ledger 2",
-			configFile:    "test/1perfile.toml",
-			start:         0,
-			end:           10,
-			expectedStart: 2,
-			expectedEnd:   10,
-		},
 		{
 			name:          "No change, 1 ledger per file",
 			configFile:    "test/1perfile.toml",
 			start:         2,
-			end:           2,
+			end:           3,
 			expectedStart: 2,
-			expectedEnd:   2,
+			expectedEnd:   3,
 		},
 		{
 			name:          "Min start ledger2, round up end ledger, 10 ledgers per file",
 			configFile:    "test/10perfile.toml",
-			start:         0,
-			end:           1,
+			start:         2,
+			end:           3,
 			expectedStart: 2,
 			expectedEnd:   9,
 		},
@@ -186,7 +364,9 @@ func TestAdjustedLedgerRangeBoundedMode(t *testing.T) {
 
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			config, err := NewConfig(Flags{StartLedger: tt.start, EndLedger: tt.end, ConfigFilePath: tt.configFile})
+			config, err := NewConfig(
+				RuntimeSettings{StartLedger: tt.start, EndLedger: tt.end, ConfigFilePath: tt.configFile, Mode: ScanFill})
+
 			require.NoError(t, err)
 			err = config.ValidateAndSetLedgerRange(ctx, mockArchive)
 			require.NoError(t, err)
@@ -194,6 +374,7 @@ func TestAdjustedLedgerRangeBoundedMode(t *testing.T) {
 			require.EqualValues(t, tt.expectedEnd, config.EndLedger)
 		})
 	}
+	mockArchive.AssertExpectations(t)
 }
 
 func TestAdjustedLedgerRangeUnBoundedMode(t *testing.T) {
@@ -205,14 +386,6 @@ func TestAdjustedLedgerRangeUnBoundedMode(t *testing.T) {
 		expectedStart uint32
 		expectedEnd   uint32
 	}{
-		{
-			name:          "Min start ledger 2",
-			configFile:    "test/1perfile.toml",
-			start:         0,
-			end:           0,
-			expectedStart: 2,
-			expectedEnd:   0,
-		},
 		{
 			name:          "No change, 1 ledger per file",
 			configFile:    "test/1perfile.toml",
@@ -254,7 +427,8 @@ func TestAdjustedLedgerRangeUnBoundedMode(t *testing.T) {
 
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			config, err := NewConfig(Flags{StartLedger: tt.start, EndLedger: tt.end, ConfigFilePath: tt.configFile})
+			config, err := NewConfig(
+				RuntimeSettings{StartLedger: tt.start, EndLedger: tt.end, ConfigFilePath: tt.configFile, Mode: Append})
 			require.NoError(t, err)
 			err = config.ValidateAndSetLedgerRange(ctx, mockArchive)
 			require.NoError(t, err)
@@ -262,6 +436,7 @@ func TestAdjustedLedgerRangeUnBoundedMode(t *testing.T) {
 			require.EqualValues(t, tt.expectedEnd, config.EndLedger)
 		})
 	}
+	mockArchive.AssertExpectations(t)
 }
 
 var cmdOut = ""
@@ -273,15 +448,20 @@ func fakeExecCommand(command string, args ...string) *exec.Cmd {
 	return cmd
 }
 
+func init() {
+	execCommand = fakeExecCommand
+}
+
 func TestExecCmdHelperProcess(t *testing.T) {
 	if os.Getenv("GO_EXEC_CMD_HELPER_PROCESS") != "1" {
 		return
 	}
-	fmt.Fprintf(os.Stdout, os.Getenv("CMD_OUT"))
+	fmt.Fprint(os.Stdout, os.Getenv("CMD_OUT"))
 	os.Exit(0)
 }
 
 func TestSetCoreVersionInfo(t *testing.T) {
+	execCommand = fakeExecCommand
 	tests := []struct {
 		name            string
 		commandOutput   string
@@ -315,9 +495,7 @@ func TestSetCoreVersionInfo(t *testing.T) {
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			config := Config{}
-
 			cmdOut = tt.commandOutput
-			execCommand = fakeExecCommand
 			err := config.setCoreVersionInfo()
 
 			if tt.expectedError != nil {
diff --git a/exp/services/ledgerexporter/internal/exportmanager_test.go b/exp/services/ledgerexporter/internal/exportmanager_test.go
index b74af2e1..7845186c 100644
--- a/exp/services/ledgerexporter/internal/exportmanager_test.go
+++ b/exp/services/ledgerexporter/internal/exportmanager_test.go
@@ -8,6 +8,7 @@ import (
 
 	"github.com/prometheus/client_golang/prometheus"
 	"github.com/stretchr/testify/mock"
+	"github.com/stretchr/testify/require"
 
 	"github.com/stretchr/testify/suite"
 
@@ -58,18 +59,18 @@ func (s *ExportManagerSuite) TearDownTest() {
 }
 
 func (s *ExportManagerSuite) TestInvalidExportConfig() {
-	config := &Config{LedgerBatchConfig: datastore.LedgerBatchConfig{LedgersPerFile: 0, FilesPerPartition: 10}}
+	config := datastore.DataStoreSchema{LedgersPerFile: 0, FilesPerPartition: 10}
 	registry := prometheus.NewRegistry()
 	queue := NewUploadQueue(1, registry)
-	_, err := NewExportManager(config, &s.mockBackend, queue, registry)
+	_, err := NewExportManager(config, &s.mockBackend, queue, registry, "passphrase", "coreversion")
 	s.Require().Error(err)
 }
 
 func (s *ExportManagerSuite) TestRun() {
-	config := &Config{LedgerBatchConfig: datastore.LedgerBatchConfig{LedgersPerFile: 64, FilesPerPartition: 10}}
+	config := datastore.DataStoreSchema{LedgersPerFile: 64, FilesPerPartition: 10}
 	registry := prometheus.NewRegistry()
 	queue := NewUploadQueue(1, registry)
-	exporter, err := NewExportManager(config, &s.mockBackend, queue, registry)
+	exporter, err := NewExportManager(config, &s.mockBackend, queue, registry, "passphrase", "coreversion")
 	s.Require().NoError(err)
 
 	start := uint32(0)
@@ -79,7 +80,7 @@ func (s *ExportManagerSuite) TestRun() {
 	for i := start; i <= end; i++ {
 		s.mockBackend.On("GetLedger", s.ctx, i).
 			Return(createLedgerCloseMeta(i), nil)
-		key := config.LedgerBatchConfig.GetObjectKeyFromSequenceNumber(i)
+		key := config.GetObjectKeyFromSequenceNumber(i)
 		expectedKeys.Add(key)
 	}
 
@@ -116,10 +117,11 @@ func (s *ExportManagerSuite) TestRun() {
 }
 
 func (s *ExportManagerSuite) TestRunContextCancel() {
-	config := &Config{LedgerBatchConfig: datastore.LedgerBatchConfig{LedgersPerFile: 1, FilesPerPartition: 1}}
+	config := datastore.DataStoreSchema{LedgersPerFile: 1, FilesPerPartition: 1}
+
 	registry := prometheus.NewRegistry()
 	queue := NewUploadQueue(1, registry)
-	exporter, err := NewExportManager(config, &s.mockBackend, queue, registry)
+	exporter, err := NewExportManager(config, &s.mockBackend, queue, registry, "passphrase", "coreversion")
 	s.Require().NoError(err)
 	ctx, cancel := context.WithCancel(context.Background())
 
@@ -146,10 +148,11 @@ func (s *ExportManagerSuite) TestRunContextCancel() {
 }
 
 func (s *ExportManagerSuite) TestRunWithCanceledContext() {
-	config := &Config{LedgerBatchConfig: datastore.LedgerBatchConfig{LedgersPerFile: 1, FilesPerPartition: 10}}
+	config := datastore.DataStoreSchema{LedgersPerFile: 1, FilesPerPartition: 10}
+
 	registry := prometheus.NewRegistry()
 	queue := NewUploadQueue(1, registry)
-	exporter, err := NewExportManager(config, &s.mockBackend, queue, registry)
+	exporter, err := NewExportManager(config, &s.mockBackend, queue, registry, "passphrase", "coreversion")
 	s.Require().NoError(err)
 
 	ctx, cancel := context.WithCancel(context.Background())
@@ -165,10 +168,11 @@ func (s *ExportManagerSuite) TestRunWithCanceledContext() {
 }
 
 func (s *ExportManagerSuite) TestAddLedgerCloseMeta() {
-	config := &Config{LedgerBatchConfig: datastore.LedgerBatchConfig{LedgersPerFile: 1, FilesPerPartition: 10}}
+	config := datastore.DataStoreSchema{LedgersPerFile: 1, FilesPerPartition: 10}
+
 	registry := prometheus.NewRegistry()
 	queue := NewUploadQueue(1, registry)
-	exporter, err := NewExportManager(config, &s.mockBackend, queue, registry)
+	exporter, err := NewExportManager(config, &s.mockBackend, queue, registry, "passphrase", "coreversion")
 	s.Require().NoError(err)
 
 	expectedKeys := set.NewSet[string](10)
@@ -193,7 +197,7 @@ func (s *ExportManagerSuite) TestAddLedgerCloseMeta() {
 	for i := start; i <= end; i++ {
 		s.Require().NoError(exporter.AddLedgerCloseMeta(context.Background(), createLedgerCloseMeta(i)))
 
-		key := config.LedgerBatchConfig.GetObjectKeyFromSequenceNumber(i)
+		key := config.GetObjectKeyFromSequenceNumber(i)
 		expectedKeys.Add(key)
 	}
 
@@ -203,10 +207,10 @@ func (s *ExportManagerSuite) TestAddLedgerCloseMeta() {
 }
 
 func (s *ExportManagerSuite) TestAddLedgerCloseMetaContextCancel() {
-	config := &Config{LedgerBatchConfig: datastore.LedgerBatchConfig{LedgersPerFile: 1, FilesPerPartition: 10}}
+	config := datastore.DataStoreSchema{LedgersPerFile: 1, FilesPerPartition: 10}
 	registry := prometheus.NewRegistry()
 	queue := NewUploadQueue(1, registry)
-	exporter, err := NewExportManager(config, &s.mockBackend, queue, registry)
+	exporter, err := NewExportManager(config, &s.mockBackend, queue, registry, "passphrase", "coreversion")
 	s.Require().NoError(err)
 
 	ctx, cancel := context.WithCancel(context.Background())
@@ -215,7 +219,7 @@ func (s *ExportManagerSuite) TestAddLedgerCloseMetaContextCancel() {
 		cancel()
 	}()
 
-	s.Require().NoError(exporter.AddLedgerCloseMeta(ctx, createLedgerCloseMeta(1)))
+	require.NoError(s.T(), exporter.AddLedgerCloseMeta(ctx, createLedgerCloseMeta(1)))
 	err = exporter.AddLedgerCloseMeta(ctx, createLedgerCloseMeta(2))
-	s.Require().EqualError(err, "context canceled")
+	require.EqualError(s.T(), err, "context canceled")
 }
diff --git a/exp/services/ledgerexporter/internal/ledger_meta_archive_test.go b/exp/services/ledgerexporter/internal/ledger_meta_archive_test.go
index 3cf263d7..152f2c64 100644
--- a/exp/services/ledgerexporter/internal/ledger_meta_archive_test.go
+++ b/exp/services/ledgerexporter/internal/ledger_meta_archive_test.go
@@ -10,10 +10,6 @@ import (
 )
 
 func TestNewLedgerMetaArchiveFromXDR(t *testing.T) {
-	config := &Config{
-		Network:     "testnet",
-		CoreVersion: "v1.2.3",
-	}
 	data := xdr.LedgerCloseMetaBatch{
 		StartSequence: 1234,
 		EndSequence:   1234,
@@ -22,7 +18,7 @@ func TestNewLedgerMetaArchiveFromXDR(t *testing.T) {
 		},
 	}
 
-	archive, err := NewLedgerMetaArchiveFromXDR(config, "key", data)
+	archive, err := NewLedgerMetaArchiveFromXDR("testnet", "v1.2.3", "key", data)
 
 	require.NoError(t, err)
 	require.NotNil(t, archive)
@@ -33,7 +29,7 @@ func TestNewLedgerMetaArchiveFromXDR(t *testing.T) {
 		EndLedger:            1234,
 		StartLedgerCloseTime: 1234 * 100,
 		EndLedgerCloseTime:   1234 * 100,
-		Network:              "testnet",
+		NetworkPassPhrase:    "testnet",
 		CompressionType:      "zstd",
 		ProtocolVersion:      21,
 		CoreVersion:          "v1.2.3",
@@ -53,7 +49,7 @@ func TestNewLedgerMetaArchiveFromXDR(t *testing.T) {
 		},
 	}
 
-	archive, err = NewLedgerMetaArchiveFromXDR(config, "key", data)
+	archive, err = NewLedgerMetaArchiveFromXDR("testnet", "v1.2.3", "key", data)
 
 	require.NoError(t, err)
 	require.NotNil(t, archive)
@@ -64,7 +60,7 @@ func TestNewLedgerMetaArchiveFromXDR(t *testing.T) {
 		EndLedger:            1237,
 		StartLedgerCloseTime: 1234 * 100,
 		EndLedgerCloseTime:   1237 * 100,
-		Network:              "testnet",
+		NetworkPassPhrase:    "testnet",
 		CompressionType:      "zstd",
 		ProtocolVersion:      21,
 		CoreVersion:          "v1.2.3",
diff --git a/exp/services/ledgerexporter/internal/main_test.go b/exp/services/ledgerexporter/internal/main_test.go
new file mode 100644
index 00000000..4c9e5412
--- /dev/null
+++ b/exp/services/ledgerexporter/internal/main_test.go
@@ -0,0 +1,116 @@
+package ledgerexporter
+
+import (
+	"bytes"
+	"io"
+	"testing"
+
+	"github.com/pkg/errors"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestFlagsOutput(t *testing.T) {
+	var testResultSettings RuntimeSettings
+	appRunnerSuccess := func(runtimeSettings RuntimeSettings) error {
+		testResultSettings = runtimeSettings
+		return nil
+	}
+
+	appRunnerError := func(runtimeSettings RuntimeSettings) error {
+		return errors.New("test error")
+	}
+
+	testCases := []struct {
+		name              string
+		commandArgs       []string
+		expectedErrOutput string
+		appRunner         func(runtimeSettings RuntimeSettings) error
+		expectedSettings  RuntimeSettings
+	}{
+		{
+			name:              "no sub-command",
+			commandArgs:       []string{"--start", "4", "--end", "5", "--config-file", "myfile"},
+			expectedErrOutput: "Error: ",
+		},
+		{
+			name:              "append sub-command with start and end present",
+			commandArgs:       []string{"append", "--start", "4", "--end", "5", "--config-file", "myfile"},
+			expectedErrOutput: "",
+			appRunner:         appRunnerSuccess,
+			expectedSettings: RuntimeSettings{
+				StartLedger:    4,
+				EndLedger:      5,
+				ConfigFilePath: "myfile",
+				Mode:           Append,
+			},
+		},
+		{
+			name:              "append sub-command with start and end absent",
+			commandArgs:       []string{"append", "--config-file", "myfile"},
+			expectedErrOutput: "",
+			appRunner:         appRunnerSuccess,
+			expectedSettings: RuntimeSettings{
+				StartLedger:    0,
+				EndLedger:      0,
+				ConfigFilePath: "myfile",
+				Mode:           Append,
+			},
+		},
+		{
+			name:              "append sub-command prints app error",
+			commandArgs:       []string{"append", "--start", "4", "--end", "5", "--config-file", "myfile"},
+			expectedErrOutput: "test error",
+			appRunner:         appRunnerError,
+		},
+		{
+			name:              "scanfill sub-command with start and end present",
+			commandArgs:       []string{"scan-and-fill", "--start", "4", "--end", "5", "--config-file", "myfile"},
+			expectedErrOutput: "",
+			appRunner:         appRunnerSuccess,
+			expectedSettings: RuntimeSettings{
+				StartLedger:    4,
+				EndLedger:      5,
+				ConfigFilePath: "myfile",
+				Mode:           ScanFill,
+			},
+		},
+		{
+			name:              "scanfill sub-command with start and end absent",
+			commandArgs:       []string{"scan-and-fill", "--config-file", "myfile"},
+			expectedErrOutput: "",
+			appRunner:         appRunnerSuccess,
+			expectedSettings: RuntimeSettings{
+				StartLedger:    0,
+				EndLedger:      0,
+				ConfigFilePath: "myfile",
+				Mode:           ScanFill,
+			},
+		},
+		{
+			name:              "scanfill sub-command prints app error",
+			commandArgs:       []string{"scan-and-fill", "--start", "4", "--end", "5", "--config-file", "myfile"},
+			expectedErrOutput: "test error",
+			appRunner:         appRunnerError,
+		},
+	}
+	for _, testCase := range testCases {
+		t.Run(testCase.name, func(t *testing.T) {
+			// mock the ledger exporter's cmd runner to be this test's mock routine instead of real app
+			ledgerExporterCmdRunner = testCase.appRunner
+			defineCommands()
+			rootCmd.SetArgs(testCase.commandArgs)
+			var errWriter io.Writer = &bytes.Buffer{}
+			var outWriter io.Writer = &bytes.Buffer{}
+			rootCmd.SetErr(errWriter)
+			rootCmd.SetOut(outWriter)
+			rootCmd.Execute()
+
+			errOutput := errWriter.(*bytes.Buffer).String()
+			if testCase.expectedErrOutput != "" {
+				assert.Contains(t, errOutput, testCase.expectedErrOutput)
+			} else {
+				assert.Equal(t, testCase.expectedSettings, testResultSettings)
+			}
+		})
+	}
+}
diff --git a/exp/services/ledgerexporter/internal/test/10perfile.toml b/exp/services/ledgerexporter/internal/test/10perfile.toml
index 9b969278..cae3d9f9 100644
--- a/exp/services/ledgerexporter/internal/test/10perfile.toml
+++ b/exp/services/ledgerexporter/internal/test/10perfile.toml
@@ -1,5 +1,6 @@
-network = "test"  
+[stellar_core_config]
+stellar_core_binary_path = "test/stellar-core"
+network = "pubnet"  
 
-[exporter_config]
-ledgers_per_file = 10
-file_suffix = ".xdr.gz"
\ No newline at end of file
+[datastore_config.schema]
+ledgers_per_file = 10
\ No newline at end of file
diff --git a/exp/services/ledgerexporter/internal/test/15perfile.toml b/exp/services/ledgerexporter/internal/test/15perfile.toml
index 94df87a2..e3e4ee9d 100644
--- a/exp/services/ledgerexporter/internal/test/15perfile.toml
+++ b/exp/services/ledgerexporter/internal/test/15perfile.toml
@@ -1,5 +1,6 @@
-network = "test"  
+[stellar_core_config]
+stellar_core_binary_path = "test/stellar-core"
+network = "pubnet"  
 
-[exporter_config]
-ledgers_per_file = 15
-file_suffix = ".xdr.gz"
\ No newline at end of file
+[datastore_config.schema]
+ledgers_per_file = 15
\ No newline at end of file
diff --git a/exp/services/ledgerexporter/internal/test/1perfile.toml b/exp/services/ledgerexporter/internal/test/1perfile.toml
index a15dc9bf..f8e88de4 100644
--- a/exp/services/ledgerexporter/internal/test/1perfile.toml
+++ b/exp/services/ledgerexporter/internal/test/1perfile.toml
@@ -1,5 +1,6 @@
-network = "test"  
+[stellar_core_config]
+stellar_core_binary_path = "test/stellar-core"
+network = "pubnet"  
 
-[exporter_config]
-ledgers_per_file = 1
-file_suffix = ".xdr.gz"
\ No newline at end of file
+[datastore_config.schema]
+ledgers_per_file = 1
\ No newline at end of file
diff --git a/exp/services/ledgerexporter/internal/test/64perfile.toml b/exp/services/ledgerexporter/internal/test/64perfile.toml
index 8e8d122f..769546af 100644
--- a/exp/services/ledgerexporter/internal/test/64perfile.toml
+++ b/exp/services/ledgerexporter/internal/test/64perfile.toml
@@ -1,5 +1,6 @@
-network = "test"  
+[stellar_core_config]
+stellar_core_binary_path = "test/stellar-core"
+network = "pubnet"  
 
-[exporter_config]
-ledgers_per_file = 64
-file_suffix = ".xdr.gz"
\ No newline at end of file
+[datastore_config.schema]
+ledgers_per_file = 64
\ No newline at end of file
diff --git a/exp/services/ledgerexporter/internal/test/captive-core-test.cfg b/exp/services/ledgerexporter/internal/test/captive-core-test.cfg
new file mode 100644
index 00000000..dac93694
--- /dev/null
+++ b/exp/services/ledgerexporter/internal/test/captive-core-test.cfg
@@ -0,0 +1,12 @@
+PEER_PORT=11725
+
+UNSAFE_QUORUM=true
+FAILURE_SAFETY=0
+
+[[VALIDATORS]]
+NAME="local_core"
+HOME_DOMAIN="core.local"
+PUBLIC_KEY="GD5KD2KEZJIGTC63IGW6UMUSMVUVG5IHG64HUTFWCHVZH2N2IBOQN7PS"
+ADDRESS="localhost"
+HISTORY="curl -sf https://localhost/{0} -o {1}"
+QUALITY="MEDIUM"
\ No newline at end of file
diff --git a/exp/services/ledgerexporter/internal/test/invalid_captive_core_toml_path.toml b/exp/services/ledgerexporter/internal/test/invalid_captive_core_toml_path.toml
new file mode 100644
index 00000000..580590b6
--- /dev/null
+++ b/exp/services/ledgerexporter/internal/test/invalid_captive_core_toml_path.toml
@@ -0,0 +1,5 @@
+[stellar_core_config]
+captive_core_toml_path = "test/notfound.cfg"
+stellar_core_binary_path = "test/stellar-core"
+history_archive_urls = ["http://testarchive"]
+network_passphrase = "test"
diff --git a/exp/services/ledgerexporter/internal/test/invalid_empty.toml b/exp/services/ledgerexporter/internal/test/invalid_empty.toml
new file mode 100644
index 00000000..8b137891
--- /dev/null
+++ b/exp/services/ledgerexporter/internal/test/invalid_empty.toml
@@ -0,0 +1,1 @@
+
diff --git a/exp/services/ledgerexporter/internal/test/invalid_preconfigured_network.toml b/exp/services/ledgerexporter/internal/test/invalid_preconfigured_network.toml
new file mode 100644
index 00000000..8e09f05e
--- /dev/null
+++ b/exp/services/ledgerexporter/internal/test/invalid_preconfigured_network.toml
@@ -0,0 +1,4 @@
+[stellar_core_config]
+stellar_core_binary_path = "test/stellar-core"
+network = "invalid"  
+
diff --git a/exp/services/ledgerexporter/internal/test/no_core_bin.toml b/exp/services/ledgerexporter/internal/test/no_core_bin.toml
new file mode 100644
index 00000000..abbd1d99
--- /dev/null
+++ b/exp/services/ledgerexporter/internal/test/no_core_bin.toml
@@ -0,0 +1,3 @@
+[stellar_core_config]
+network = "testnet"  
+ 
\ No newline at end of file
diff --git a/exp/services/ledgerexporter/internal/test/no_network.toml b/exp/services/ledgerexporter/internal/test/no_network.toml
deleted file mode 100644
index 1cb591cd..00000000
--- a/exp/services/ledgerexporter/internal/test/no_network.toml
+++ /dev/null
@@ -1,11 +0,0 @@
-
-[datastore_config]
-type = "ABC"
-
-[datastore_config.params]
-destination_bucket_path = "your-bucket-name/subpath"
-
-[exporter_config]
-ledgers_per_file = 3
-files_per_partition = 1
-file_suffix = ".xdr.gz"
\ No newline at end of file
diff --git a/exp/services/ledgerexporter/internal/test/test.toml b/exp/services/ledgerexporter/internal/test/test.toml
index 58b5fc6d..6a6804c7 100644
--- a/exp/services/ledgerexporter/internal/test/test.toml
+++ b/exp/services/ledgerexporter/internal/test/test.toml
@@ -1,12 +1,13 @@
-network = "test"  
+[stellar_core_config]
+stellar_core_binary_path = "test/stellar-core"
+network = "pubnet"  
 
 [datastore_config]
 type = "ABC"
 
 [datastore_config.params]
-destination_bucket_path = "your-bucket-name/subpath"
+destination_bucket_path = "your-bucket-name/subpath/testnet"
 
-[exporter_config]
+[datastore_config.schema]
 ledgers_per_file = 3
-files_per_partition = 1
-file_suffix = ".xdr.gz"
\ No newline at end of file
+files_per_partition = 1
\ No newline at end of file
diff --git a/exp/services/ledgerexporter/internal/test/useragent.toml b/exp/services/ledgerexporter/internal/test/useragent.toml
new file mode 100644
index 00000000..209c0415
--- /dev/null
+++ b/exp/services/ledgerexporter/internal/test/useragent.toml
@@ -0,0 +1,7 @@
+user_agent = "useragent_x"
+
+[stellar_core_config]
+stellar_core_binary_path = "test/stellar-core"
+network = "pubnet"  
+
+
diff --git a/exp/services/ledgerexporter/internal/test/valid_captive_core_manual.toml b/exp/services/ledgerexporter/internal/test/valid_captive_core_manual.toml
new file mode 100644
index 00000000..24a43048
--- /dev/null
+++ b/exp/services/ledgerexporter/internal/test/valid_captive_core_manual.toml
@@ -0,0 +1,5 @@
+[stellar_core_config]
+captive_core_toml_path = "test/captive-core-test.cfg"
+stellar_core_binary_path = "test/stellar-core"
+history_archive_urls = ["http://testarchive"]
+network_passphrase = "test"
\ No newline at end of file
diff --git a/exp/services/ledgerexporter/internal/test/valid_captive_core_override.toml b/exp/services/ledgerexporter/internal/test/valid_captive_core_override.toml
new file mode 100644
index 00000000..312ccb29
--- /dev/null
+++ b/exp/services/ledgerexporter/internal/test/valid_captive_core_override.toml
@@ -0,0 +1,4 @@
+[stellar_core_config]
+network="pubnet"
+captive_core_toml_path = "test/captive-core-test.cfg"
+stellar_core_binary_path = "test/stellar-core"
diff --git a/exp/services/ledgerexporter/internal/test/valid_captive_core_override_archives.toml b/exp/services/ledgerexporter/internal/test/valid_captive_core_override_archives.toml
new file mode 100644
index 00000000..9aebc153
--- /dev/null
+++ b/exp/services/ledgerexporter/internal/test/valid_captive_core_override_archives.toml
@@ -0,0 +1,4 @@
+[stellar_core_config]
+network="pubnet"
+history_archive_urls = ["http://testarchive"]
+stellar_core_binary_path = "test/stellar-core"
diff --git a/exp/services/ledgerexporter/internal/test/valid_captive_core_preconfigured.toml b/exp/services/ledgerexporter/internal/test/valid_captive_core_preconfigured.toml
new file mode 100644
index 00000000..cb1c3c5c
--- /dev/null
+++ b/exp/services/ledgerexporter/internal/test/valid_captive_core_preconfigured.toml
@@ -0,0 +1,4 @@
+[stellar_core_config]
+stellar_core_binary_path = "test/stellar-core"
+network = "pubnet"  
+
diff --git a/exp/services/ledgerexporter/internal/test/validate_start_end.toml b/exp/services/ledgerexporter/internal/test/validate_start_end.toml
index a15dc9bf..8cbd9f37 100644
--- a/exp/services/ledgerexporter/internal/test/validate_start_end.toml
+++ b/exp/services/ledgerexporter/internal/test/validate_start_end.toml
@@ -1,5 +1,6 @@
-network = "test"  
-
-[exporter_config]
+[datastore_config.schema]
 ledgers_per_file = 1
-file_suffix = ".xdr.gz"
\ No newline at end of file
+
+[stellar_core_config]
+stellar_core_binary_path = "test/stellar-core"
+network = "pubnet"  
\ No newline at end of file
diff --git a/exp/services/ledgerexporter/internal/uploader_test.go b/exp/services/ledgerexporter/internal/uploader_test.go
index bb3ca1e2..612c9e87 100644
--- a/exp/services/ledgerexporter/internal/uploader_test.go
+++ b/exp/services/ledgerexporter/internal/uploader_test.go
@@ -59,7 +59,7 @@ func (s *UploaderSuite) TestUploadWithMetadata() {
 		EndLedgerCloseTime:   987654321,
 		ProtocolVersion:      3,
 		CoreVersion:          "v1.2.3",
-		Network:              "testnet",
+		NetworkPassPhrase:    "testnet",
 		CompressionType:      "gzip",
 		Version:              "1.0.0",
 	}
diff --git a/ingest/ledgerbackend/buffered_storage_backend_test.go b/ingest/ledgerbackend/buffered_storage_backend_test.go
index 1aca8306..f18329ff 100644
--- a/ingest/ledgerbackend/buffered_storage_backend_test.go
+++ b/ingest/ledgerbackend/buffered_storage_backend_test.go
@@ -44,7 +44,7 @@ func createBufferedStorageBackendConfigForTesting() BufferedStorageBackendConfig
 	param := make(map[string]string)
 	param["destination_bucket_path"] = "testURL"
 
-	ledgerBatchConfig := datastore.LedgerBatchConfig{
+	ledgerBatchConfig := datastore.DataStoreSchema{
 		LedgersPerFile:    1,
 		FilesPerPartition: 64000,
 	}
diff --git a/support/datastore/datastore_test.go b/support/datastore/datastore_test.go
index 12041729..3482ef32 100644
--- a/support/datastore/datastore_test.go
+++ b/support/datastore/datastore_test.go
@@ -8,6 +8,6 @@ import (
 )
 
 func TestInvalidStore(t *testing.T) {
-	_, err := NewDataStore(context.Background(), DataStoreConfig{Type: "unknown"}, "test")
+	_, err := NewDataStore(context.Background(), DataStoreConfig{Type: "unknown"})
 	require.Error(t, err)
 }
diff --git a/support/datastore/gcs_test.go b/support/datastore/gcs_test.go
index 0ddf60ac..8838e8da 100644
--- a/support/datastore/gcs_test.go
+++ b/support/datastore/gcs_test.go
@@ -24,7 +24,7 @@ func TestGCSExists(t *testing.T) {
 	})
 	defer server.Stop()
 
-	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects", "testnet")
+	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects/testnet")
 	require.NoError(t, err)
 	t.Cleanup(func() {
 		require.NoError(t, store.Close())
@@ -52,7 +52,7 @@ func TestGCSSize(t *testing.T) {
 	})
 	defer server.Stop()
 
-	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects", "testnet")
+	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects/testnet")
 	require.NoError(t, err)
 	t.Cleanup(func() {
 		require.NoError(t, store.Close())
@@ -86,7 +86,7 @@ func TestGCSPutFile(t *testing.T) {
 		DefaultEventBasedHold: false,
 	})
 
-	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects", "testnet")
+	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects/testnet")
 	require.NoError(t, err)
 	t.Cleanup(func() {
 		require.NoError(t, store.Close())
@@ -138,7 +138,7 @@ func TestGCSPutFileIfNotExists(t *testing.T) {
 	})
 	defer server.Stop()
 
-	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects", "testnet")
+	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects/testnet")
 	require.NoError(t, err)
 	t.Cleanup(func() {
 		require.NoError(t, store.Close())
@@ -187,7 +187,7 @@ func TestGCSPutFileWithMetadata(t *testing.T) {
 		DefaultEventBasedHold: false,
 	})
 
-	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects", "testnet")
+	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects/testnet")
 	require.NoError(t, err)
 	t.Cleanup(func() {
 		require.NoError(t, store.Close())
@@ -198,7 +198,7 @@ func TestGCSPutFileWithMetadata(t *testing.T) {
 		EndLedger:            1234,
 		StartLedgerCloseTime: 1234,
 		EndLedgerCloseTime:   1234,
-		Network:              "testnet",
+		NetworkPassPhrase:    "testnet",
 		CompressionType:      "zstd",
 		ProtocolVersion:      21,
 		CoreVersion:          "v1.2.3",
@@ -223,7 +223,7 @@ func TestGCSPutFileWithMetadata(t *testing.T) {
 		EndLedger:            6789,
 		StartLedgerCloseTime: 1622547800,
 		EndLedgerCloseTime:   1622548900,
-		Network:              "mainnet",
+		NetworkPassPhrase:    "mainnet",
 		CompressionType:      "gzip",
 		ProtocolVersion:      23,
 		CoreVersion:          "v1.4.0",
@@ -255,7 +255,7 @@ func TestGCSPutFileIfNotExistsWithMetadata(t *testing.T) {
 	})
 	defer server.Stop()
 
-	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects", "testnet")
+	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects/testnet")
 	require.NoError(t, err)
 	t.Cleanup(func() {
 		require.NoError(t, store.Close())
@@ -265,7 +265,7 @@ func TestGCSPutFileIfNotExistsWithMetadata(t *testing.T) {
 		EndLedger:            1234,
 		StartLedgerCloseTime: 1234,
 		EndLedgerCloseTime:   1234,
-		Network:              "testnet",
+		NetworkPassPhrase:    "testnet",
 		CompressionType:      "zstd",
 		ProtocolVersion:      21,
 		CoreVersion:          "v1.2.3",
@@ -290,7 +290,7 @@ func TestGCSPutFileIfNotExistsWithMetadata(t *testing.T) {
 		EndLedger:            6789,
 		StartLedgerCloseTime: 1622547800,
 		EndLedgerCloseTime:   1622548900,
-		Network:              "mainnet",
+		NetworkPassPhrase:    "mainnet",
 		CompressionType:      "gzip",
 		ProtocolVersion:      23,
 		CoreVersion:          "v1.4.0",
@@ -323,7 +323,7 @@ func TestGCSGetNonExistentFile(t *testing.T) {
 	})
 	defer server.Stop()
 
-	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects", "testnet")
+	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects/testnet")
 	require.NoError(t, err)
 	t.Cleanup(func() {
 		require.NoError(t, store.Close())
@@ -365,7 +365,7 @@ func TestGCSGetFileValidatesCRC32C(t *testing.T) {
 	})
 	defer server.Stop()
 
-	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects", "testnet")
+	store, err := FromGCSClient(context.Background(), server.Client(), "test-bucket/objects/testnet")
 	require.NoError(t, err)
 	t.Cleanup(func() {
 		require.NoError(t, store.Close())
diff --git a/support/datastore/ledgerbatch_config_test.go b/support/datastore/ledgerbatch_config_test.go
index bd79dd26..f95fef16 100644
--- a/support/datastore/ledgerbatch_config_test.go
+++ b/support/datastore/ledgerbatch_config_test.go
@@ -31,7 +31,7 @@ func TestGetObjectKeyFromSequenceNumber(t *testing.T) {
 
 	for _, tc := range testCases {
 		t.Run(fmt.Sprintf("LedgerSeq-%d-LedgersPerFile-%d", tc.ledgerSeq, tc.ledgersPerFile), func(t *testing.T) {
-			config := LedgerBatchConfig{FilesPerPartition: tc.filesPerPartition, LedgersPerFile: tc.ledgersPerFile}
+			config := DataStoreSchema{FilesPerPartition: tc.filesPerPartition, LedgersPerFile: tc.ledgersPerFile}
 			key := config.GetObjectKeyFromSequenceNumber(tc.ledgerSeq)
 			require.Equal(t, tc.expectedKey, key)
 		})
@@ -39,7 +39,7 @@ func TestGetObjectKeyFromSequenceNumber(t *testing.T) {
 }
 
 func TestGetObjectKeyFromSequenceNumber_ObjectKeyDescOrder(t *testing.T) {
-	config := LedgerBatchConfig{
+	config := DataStoreSchema{
 		LedgersPerFile:    1,
 		FilesPerPartition: 10,
 	}
diff --git a/support/datastore/metadata_test.go b/support/datastore/metadata_test.go
index 6ec6a176..ffc92f4e 100644
--- a/support/datastore/metadata_test.go
+++ b/support/datastore/metadata_test.go
@@ -22,7 +22,7 @@ func TestMetaDataToMap(t *testing.T) {
 				EndLedgerCloseTime:   987654321,
 				ProtocolVersion:      3,
 				CoreVersion:          "v1.2.3",
-				Network:              "testnet",
+				NetworkPassPhrase:    "testnet passphrase",
 				CompressionType:      "gzip",
 				Version:              "1.0.0",
 			},
@@ -33,7 +33,7 @@ func TestMetaDataToMap(t *testing.T) {
 				"end-ledger-close-time":   "987654321",
 				"protocol-version":        "3",
 				"core-version":            "v1.2.3",
-				"network":                 "testnet",
+				"network-passphrase":      "testnet passphrase",
 				"compression-type":        "gzip",
 				"version":                 "1.0.0",
 			},
@@ -48,7 +48,7 @@ func TestMetaDataToMap(t *testing.T) {
 				EndLedgerCloseTime:   tt.metaData.EndLedgerCloseTime,
 				ProtocolVersion:      tt.metaData.ProtocolVersion,
 				CoreVersion:          tt.metaData.CoreVersion,
-				Network:              tt.metaData.Network,
+				NetworkPassPhrase:    tt.metaData.NetworkPassPhrase,
 				CompressionType:      tt.metaData.CompressionType,
 				Version:              tt.metaData.Version,
 			}
@@ -66,7 +66,7 @@ func TestNewMetaDataFromMap(t *testing.T) {
 		"end-ledger-close-time":   "987654321",
 		"protocol-version":        "3",
 		"core-version":            "v1.2.3",
-		"network":                 "testnet",
+		"network-passphrase":      "testnet passphrase",
 		"compression-type":        "gzip",
 		"version":                 "1.0.0",
 	}
@@ -78,7 +78,7 @@ func TestNewMetaDataFromMap(t *testing.T) {
 		EndLedgerCloseTime:   987654321,
 		ProtocolVersion:      3,
 		CoreVersion:          "v1.2.3",
-		Network:              "testnet",
+		NetworkPassPhrase:    "testnet passphrase",
 		CompressionType:      "gzip",
 		Version:              "1.0.0",
 	}
diff --git a/support/datastore/resumablemanager_test.go b/support/datastore/resumablemanager_test.go
index 2649a260..4616f9e4 100644
--- a/support/datastore/resumablemanager_test.go
+++ b/support/datastore/resumablemanager_test.go
@@ -16,10 +16,9 @@ func TestResumability(t *testing.T) {
 		name              string
 		startLedger       uint32
 		endLedger         uint32
-		ledgerBatchConfig LedgerBatchConfig
+		dataStoreSchema   DataStoreSchema
 		absentLedger      uint32
 		findStartOk       bool
-		networkName       string
 		latestLedger      uint32
 		errorSnippet      string
 		archiveError      error
@@ -31,11 +30,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    0,
 			absentLedger: 0,
 			findStartOk:  false,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(1),
 				LedgersPerFile:    uint32(10),
 			},
-			networkName:       "test",
 			errorSnippet:      "archive error",
 			archiveError:      errors.New("archive error"),
 			registerMockCalls: func(store *MockDataStore) {},
@@ -46,11 +44,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    4,
 			absentLedger: 0,
 			findStartOk:  false,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(1),
 				LedgersPerFile:    uint32(10),
 			},
-			networkName: "test",
 			registerMockCalls: func(mockDataStore *MockDataStore) {
 				mockDataStore.On("Exists", ctx, "FFFFFFFF--0-9.xdr.zstd").Return(true, nil).Once()
 			},
@@ -61,11 +58,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    14,
 			absentLedger: 14,
 			findStartOk:  true,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(1),
 				LedgersPerFile:    uint32(10),
 			},
-			networkName: "test",
 			registerMockCalls: func(mockDataStore *MockDataStore) {
 				mockDataStore.On("Exists", ctx, "FFFFFFF5--10-19.xdr.zstd").Return(false, nil).Twice()
 			},
@@ -76,11 +72,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    68,
 			absentLedger: 64,
 			findStartOk:  true,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(100),
 				LedgersPerFile:    uint32(64),
 			},
-			networkName: "test",
 			registerMockCalls: func(mockDataStore *MockDataStore) {
 				mockDataStore.On("Exists", ctx, "FFFFFFFF--0-6399/FFFFFFBF--64-127.xdr.zstd").Return(false, nil).Twice()
 			},
@@ -91,11 +86,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    130,
 			absentLedger: 0,
 			findStartOk:  false,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(100),
 				LedgersPerFile:    uint32(64),
 			},
-			networkName: "test",
 			registerMockCalls: func(mockDataStore *MockDataStore) {
 				mockDataStore.On("Exists", ctx, "FFFFFFFF--0-6399/FFFFFF7F--128-191.xdr.zstd").Return(true, nil).Once()
 			},
@@ -106,11 +100,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    127,
 			absentLedger: 2,
 			findStartOk:  true,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(100),
 				LedgersPerFile:    uint32(64),
 			},
-			networkName: "test",
 			registerMockCalls: func(mockDataStore *MockDataStore) {
 				mockDataStore.On("Exists", ctx, "FFFFFFFF--0-6399/FFFFFFBF--64-127.xdr.zstd").Return(true, nil).Once()
 				mockDataStore.On("Exists", ctx, "FFFFFFFF--0-6399/FFFFFFFF--0-63.xdr.zstd").Return(false, nil).Once()
@@ -122,11 +115,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    24,
 			absentLedger: 0,
 			findStartOk:  false,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(1),
 				LedgersPerFile:    uint32(10),
 			},
-			networkName:  "test",
 			errorSnippet: "datastore error happened",
 			registerMockCalls: func(mockDataStore *MockDataStore) {
 				mockDataStore.On("Exists", ctx, "FFFFFFEB--20-29.xdr.zstd").Return(false, errors.New("datastore error happened")).Once()
@@ -138,11 +130,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    50,
 			absentLedger: 40,
 			findStartOk:  true,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(1),
 				LedgersPerFile:    uint32(10),
 			},
-			networkName: "test",
 			registerMockCalls: func(mockDataStore *MockDataStore) {
 				mockDataStore.On("Exists", ctx, "FFFFFFCD--50-59.xdr.zstd").Return(false, nil).Once()
 				mockDataStore.On("Exists", ctx, "FFFFFFE1--30-39.xdr.zstd").Return(true, nil).Once()
@@ -155,11 +146,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    85,
 			absentLedger: 80,
 			findStartOk:  true,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(1),
 				LedgersPerFile:    uint32(10),
 			},
-			networkName: "test",
 			registerMockCalls: func(mockDataStore *MockDataStore) {
 				mockDataStore.On("Exists", ctx, "FFFFFFB9--70-79.xdr.zstd").Return(true, nil).Once()
 				mockDataStore.On("Exists", ctx, "FFFFFFAF--80-89.xdr.zstd").Return(false, nil).Twice()
@@ -171,11 +161,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    275,
 			absentLedger: 0,
 			findStartOk:  false,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(1),
 				LedgersPerFile:    uint32(10),
 			},
-			networkName: "test",
 			registerMockCalls: func(mockDataStore *MockDataStore) {
 				mockDataStore.On("Exists", ctx, "FFFFFEFB--260-269.xdr.zstd").Return(true, nil).Once()
 				mockDataStore.On("Exists", ctx, "FFFFFEF1--270-279.xdr.zstd").Return(true, nil).Once()
@@ -187,11 +176,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    125,
 			absentLedger: 95,
 			findStartOk:  true,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(1),
 				LedgersPerFile:    uint32(10),
 			},
-			networkName: "test",
 			registerMockCalls: func(mockDataStore *MockDataStore) {
 				mockDataStore.On("Exists", ctx, "FFFFFF87--120-129.xdr.zstd").Return(false, nil).Once()
 				mockDataStore.On("Exists", ctx, "FFFFFF91--110-119.xdr.zstd").Return(false, nil).Once()
@@ -205,11 +193,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    10,
 			absentLedger: 0,
 			findStartOk:  false,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(1),
 				LedgersPerFile:    uint32(10),
 			},
-			networkName:       "test",
 			errorSnippet:      "Invalid start value",
 			registerMockCalls: func(store *MockDataStore) {},
 		},
@@ -219,11 +206,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    0,
 			absentLedger: 1145,
 			findStartOk:  true,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(1),
 				LedgersPerFile:    uint32(10),
 			},
-			networkName:  "test2",
 			latestLedger: uint32(2000),
 			registerMockCalls: func(mockDataStore *MockDataStore) {
 				mockDataStore.On("Exists", ctx, "FFFFF9A1--1630-1639.xdr.zstd").Return(false, nil).Once()
@@ -242,11 +228,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    0,
 			absentLedger: 2250,
 			findStartOk:  true,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(1),
 				LedgersPerFile:    uint32(10),
 			},
-			networkName:  "test3",
 			latestLedger: uint32(3000),
 			registerMockCalls: func(mockDataStore *MockDataStore) {
 				mockDataStore.On("Exists", ctx, "FFFFF5B9--2630-2639.xdr.zstd").Return(false, nil).Once()
@@ -264,11 +249,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    0,
 			absentLedger: 4070,
 			findStartOk:  true,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(1),
 				LedgersPerFile:    uint32(10),
 			},
-			networkName:  "test4",
 			latestLedger: uint32(4000),
 			registerMockCalls: func(mockDataStore *MockDataStore) {
 				mockDataStore.On("Exists", ctx, "FFFFF1D1--3630-3639.xdr.zstd").Return(true, nil).Once()
@@ -286,11 +270,10 @@ func TestResumability(t *testing.T) {
 			endLedger:    0,
 			absentLedger: 0,
 			findStartOk:  false,
-			ledgerBatchConfig: LedgerBatchConfig{
+			dataStoreSchema: DataStoreSchema{
 				FilesPerPartition: uint32(1),
 				LedgersPerFile:    uint32(10),
 			},
-			networkName:       "test5",
 			latestLedger:      uint32(5000),
 			errorSnippet:      "Invalid start value of 5129, it is greater than network's latest ledger of 5128",
 			registerMockCalls: func(store *MockDataStore) {},
@@ -304,7 +287,7 @@ func TestResumability(t *testing.T) {
 			mockDataStore := &MockDataStore{}
 			tt.registerMockCalls(mockDataStore)
 
-			resumableManager := NewResumableManager(mockDataStore, tt.networkName, tt.ledgerBatchConfig, mockArchive)
+			resumableManager := NewResumableManager(mockDataStore, tt.dataStoreSchema, mockArchive)
 			absentLedger, ok, err := resumableManager.FindStart(ctx, tt.startLedger, tt.endLedger)
 			if tt.errorSnippet != "" {
 				require.ErrorContains(t, err, tt.errorSnippet)