Repository: cloudspannerecosystem/harbourbridge
Pull request #208 (2021-10-27 22:06:08)

Add support for source-profile and target-profile in subcommands.
This PR introduces support for specifying various options for specifying source and target through 2 new flags: `-source-profile` and `-target-profile`. Both of these flags a list of key value pairs as specified in the example usage below. 

For backward compatibility, we provide helper functions to convert SourceProfile and TargetProfile to legacy flags `-driver` and `-target-db`, but eventually we will remove usage of these legacy flags and SourceProfile and TargetProfile structs will be passed throughout the codebase consistently.

This PR also contains updated copies of various READMEs which we will keep updating with new subcommands and flags replace the original READMEs on master once we are ready to switch to new command line interface.

Example usage with new flags structure:

# Specifying source profile

## PostgreSQL dump

```sh
pg_dump mydb | harbourbridge schema -source=postgresql

harbourbridge schema -source=postgres < my_pg_dump_file

# Can specify file format explicitly, which is dump by default and is the only supported format today.

harbourbridge schema -source=postgresql -source-profile="format=dump,file=<full path to dump file>"

harbourbridge schema -source=postgresql -source-profile="format=dump,file=gs://<bucket_name>/<file_path>"
```

## PostgreSQL direct connect

If dump file is not specified on the stdin or through flag in source-profile, assume direct connect.

```sh
harbourbridge schema -source=postgres
```

## MySQL dump

```sh
mysqldump mydb | harbourbridge schema -source=mysql

harbourbridge schema -source=mysql < my_mysqldump_file

harbourbridge schema -prefix=mydb. -source=mysql < my_mysqldump_file
```

## MySQL direct connect

If dump file is not specified on the stdin or through flag in source-profile, assume direct connect.

```sh
export MYSQLHOST=<>
export MYSQLPORT=<>
export MYSQLUSER=<>
export MYSQLDATABASE=<>
harbourbridge schema -source=mysql
```

## DynamoDB
```sh
harbourbridge schema -source=dynamodb

harbourbridge schema -source=dynamodb -source-profile="schema-sample-size=500000"
```

# Specifying target profile

Default target is Spanner, which is also the only supported target database today.
```sh
harbourbridge schema -source=mysql -target=spanner < my_mysqldump_file
```

To specify PG Spanner database, dialect can be specified
```sh
harbourbridge schema -source=mysql -target-profile="dialect=postgresql" < my_mysqldump_file
```

Spanner instance name can be specified

```sh
harbourbridge schema -source=mysql -target-profile="instance=my-spanner-instance" < my_mysqldump_file
```

Spanner database name can be specified

```sh
harbourbridge schema -source=mysql -target-profile="dbname=my-spanner-database-name" < my_mysqldump_file
```



Patch:
diff --git a/.gitignore b/.gitignore
index 64055d5..da3e1d4 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,3 +2,4 @@
 main
 web/harbour_bridge_output/
 *__debug_bin*
+*.code-workspace
\ No newline at end of file
diff --git a/cmd/README.dynamodb.md b/cmd/README.dynamodb.md
new file mode 100644
index 0000000..881b95b
--- /dev/null
+++ b/cmd/README.dynamodb.md
@@ -0,0 +1,147 @@
+# HarbourBridge: Turnkey DynamoDB-to-Spanner Evaluation
+
+HarbourBridge is a stand-alone open source tool for Cloud Spanner evaluation,
+using data from an existing PostgreSQL or MySQL or DynamoDB database. This
+README provides details of the tool's DynamoDB capabilities. For general
+HarbourBridge information see this [README](https://github.com/cloudspannerecosystem/harbourbridge#harbourbridge-turnkey-spanner-evaluation).
+
+## Example DynamoDB Usage
+
+The following examples assume `harbourbridge` has been added to your PATH
+environment variable.
+
+Before running HarbourBridge, make sure that you have
+[set up your AWS credentials/region](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html)
+correctly (set the environment variables `AWS_ACCESS_KEY_ID`, 
+`AWS_SECRET_ACCESS_KEY`, `AWS_REGION`). HarbourBridge accesses your 
+DynamoDB database via the aws go sdk package. If you use a custom endpoint
+for dynamodb, you can specify that using the environment variable 
+`DYNAMODB_ENDPOINT_OVERRIDE`.
+
+```sh
+export DYNAMODB_ENDPOINT_OVERRIDE=http://dynamodb.us-west-2.amazonaws.com`
+```
+
+To use the tool on a DynamoDB database (it will migrate all tables),
+run
+
+```sh
+harbourbridge schema -source=dynamodb
+```
+
+### Source profile (`-source-profile`)
+
+HarbourBridge accepts the following options for --source-profile for DynamoDB,
+specified as "key1=value1,key2=value,..." pairs:
+
+`-schema-sample-size` Due to the schemaless nature of DynamoDB, the tool infers
+the schema based on a certain amount of sampled data, by default, 100000 rows.
+If a table has more rows than the default value, we only use 100000 rows for 
+estimating the schema. This flag lets you specify the number of rows to use for
+inferring schema. The default value is 100,000.
+
+```sh
+harbourbridge schema -source=dynamodb -source-profile="schema-sample-size=500000"
+```
+
+## Schema Conversion
+
+The HarbourBridge tool maps DynamoDB types to Spanner types as follows:
+
+| PostgreSQL Type    | Spanner Type               | Notes                                     |
+| ------------------ | -------------------------- | ----------------------------------------- |
+| `Number`           | `NUMERIC` or `STRING`      | defaults to NUMERIC, otherwise, STRING    |
+| `String`           | `STRING`                   |                                           |
+| `Boolean`          | `BOOL`                     |                                           |
+| `Binary`           | `BYTES`                    |                                           |
+| `Null`             | A nullable column type     |                                           |
+| `List`             | `STRING`                   | json encoding                             |
+| `Map`              | `STRING`                   | json encoding                             |
+| `StringSet`        | `ARRAY<STRING>`            |                                           |
+| `NumberSet`        | `ARRAY<NUMERIC or STRING>` |                                           |
+| `BinarySet`        | `ARRAY<BYTES>`             |                                           |
+
+We discuss these, as well as other limits and notes on schema conversion, in the
+following sections.
+
+### Schema Inference
+
+DynamoDB is a schemaless database: other than a primary index and optional
+secondary index, column names and types are essentially unconstrained
+and can vary from one row to the next.
+
+However, many customers use DynamoDB in a consistent, structured way
+with a fairly well defined set of columns and types. Our HarbourBridge support
+for DynamoDB focuses on this use-case, and we construct a Spanner schema
+by inspecting table data.
+
+For small tables, we inspect all rows of the table. For large tables, scanning
+the entire table would be extremely expensive and slow, and so we only inspect
+the first N rows (defined by the flag `schema-sample-size`) from the table scan.
+While DynamoDB doesn't return scan results in order, they might not be a truly
+random sample of rows. However, the alternative of randomly sampling rows
+would be much more expensive.
+
+Columns with consistent types are assigned Spanner types as detailed below.
+Columns without a consistent type are mapped to STRING.
+
+#### `Number`
+
+In most cases, we map the Number type in DynamoDB to Spanner's Numeric type.
+However, since the [range of Numeric](https://cloud.google.com/spanner/docs/storing-numeric-data)
+in Cloud Spanner is smaller than the [range of Number](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html)
+in DynamoDB, this conversion could result in out of range with potential
+precision loss. To address this possibility, we try to convert the sample data,
+and if it consistently fails, we choose STRING type for the column.
+
+#### `Null` Data Type
+
+In DynamoDB, a column can have a Null data type that represents an unknown or
+undefined state. Also, each row defines its own schema for columns (not for
+primary keys). So columns can be absent in rows. 
+
+We treat the above two cases the same as a Null value in Cloud Spanner. The
+cases that a column contains a Null value or a column is not present is an
+indication that this column should be nullable.
+
+#### `List` and `Map`
+
+In Cloud Spanner, the most similar type to List and Map is
+[STRUCT](https://cloud.google.com/spanner/docs/data-types#struct_type), but it
+is not a valid column type (available for query but not for storage).
+Therefore, we encode them into a json string. 
+
+#### Occasional Errors
+
+To prevent a few spurious rows from impacting schema construction, we define an
+error threshold: when building a type for a column, if the percentage of rows
+with a specific data type is lower than or equal to an extremely low
+value (0.1%), then we treat those rows as likely errors. Such rows are ignored
+for schema construction: their type is not considered a candidate type for the
+column.
+
+#### Multi-type Columns
+
+For a special scenario, we may get a column that has equal distribution of two
+data types. E.g., a column has 40% rows in String and 60% rows in Number. If we
+choose Number as its type, it may lead to 40% data loss in the data conversion.
+In the migration, we define a conflicting threshold on rows (after removing Null
+data types and rows that the column is not present). By default, the conflicting
+threshold is 5% and if the percentages of two or more data types are greater
+than it, we would consider that the column has conflicting data types. As a safe
+choice, we define this column as a STRING type in Cloud Spanner. 
+
+## Data Conversion
+
+### A Scan for Entire Table
+
+Data conversion proceeds table by table. For each table, we use the Scan API to
+read data. Each read has a size limit up to 1MB. By using the returned token, we
+make a subsequent call to continue retrieving data from the table.
+
+The row result contains the data type and data itself. According to our
+[inferred schema](#schema-inference), we will parse the row to a format that
+Cloud Spanner can support. If the value parsing fails, we would drop the entire
+row and record it as bad data in the report. If a column does not appear or 
+column has a NULL data type, we would process this as a NULL value in 
+Cloud Spanner. 
diff --git a/cmd/README.md b/cmd/README.md
new file mode 100644
index 0000000..3c9cc05
--- /dev/null
+++ b/cmd/README.md
@@ -0,0 +1,527 @@
+# HarbourBridge: Spanner Evaluation and Migration
+
+[![cloudspannerecosystem](https://circleci.com/gh/cloudspannerecosystem/harbourbridge.svg?style=svg)](https://circleci.com/gh/cloudspannerecosystem/harbourbridge)
+
+HarbourBridge is a stand-alone open source tool for Cloud Spanner evaluation and
+migration, using data from an existing PostgreSQL, MySQL or DynamoDB database. 
+The tool ingests schema and data from either a pg_dump/mysqldump file or directly 
+from the source database, and supports both schema and data migration. For schema
+migration, HarbourBridge automatically builds a Spanner schema from the schema
+of the source database. This schema can be customized using the HarbourBridge
+schema assistant. For data migration, HarbourBridge creates a new Spanner
+database using the Spanner schema built during schema migration, and populates
+it with data from the source database.
+
+For more details on schema customization and use of the schema assistant, see
+[web/README](web/README.md). The rest of this README describes the command-line
+capabilities of HarbourBridge.
+
+HarbourBridge is designed to simplify Spanner evaluation and migration, and in
+particular for migrating moderate-size PostgreSQL/MySQL/DynamoDB datasets to Spanner 
+(up to about 100GB). Many features of PostgreSQL/MySQL, especially those that don't
+map directly to Spanner features, are ignored, e.g. stored functions and
+procedures, and sequences. Types such as integers, floats, char/text, bools,
+timestamps, and (some) array types, map fairly directly to Spanner, but many
+other types do not and instead are mapped to Spanner's `STRING(MAX)`.
+In the case of DynamoDB, the schema is inferred based on a certain 
+amount of sampled data.
+
+View HarbourBridge as a way to get up and running fast, so you can focus on
+critical things like tuning performance and getting the most out of
+Spanner. Expect that you'll need to tweak and enhance what HarbourBridge
+produces.
+
+Below are some quick starter examples on how to run HarbourBridge. You can look
+at specific sections for PostgreSQL, MySQL etc for detailed examples.
+
+For example, to generate Spanner schema from a PostgreSQL dump file, run
+
+```sh
+# Generate pg_dump file.
+pg_dump mydb > mydb.pg_dump
+
+harbourbridge schema -source=postgresql -source-profile="format=dump" < mydb.pg_dump
+# Or, (By default, format is "dump" when specifying input files)
+harbourbridge schema -source=postgresql < mydb.pg_dump
+# Or, you can specify full path to the dump file
+harbourbridge schema -source=postgresql -source-profile="format=dump,file=<full path to dump file>"
+# You can also specify path to a file stored in GCS (Google Cloud Storage)
+harbourbridge schema -source=postgresql -source-profile="format=dump,file=gs://<bucket_name>/<file_path>"
+
+# Target is Spanner by default, but you can also specify it explicitly.
+harbourbridge schema -source=postgresql -source-profile="format=dump" -target=spanner < mydb.pg_dump
+```
+
+Similarly, you can use the tool for other source databases.
+
+```sh
+# Generate Spanner schema from a MySQL dump file
+mysqldump mydb > mydb.mysqldump
+harbourbridge schema -source=mysql < mydb.mysqldump
+```
+
+Generate Spanner schema by directly connecting to a DynamoDB database,
+```sh
+# Set the environment variables `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
+# , `AWS_REGION` and then run,
+harbourbridge schema -source=dynamodb
+```
+
+Note: HarbourBridge accepts pg_dump/mysqldump's standard plain-text format, 
+but not archive or custom formats. More details on usage can be found in 
+[Example usage](#example-usage) section.
+
+HarbourBridge automatically determines the cloud project to use, and generates
+a new Spanner database name (prefixed with `{source}_` and today's date).
+Command-line flags can be used to explicitly set the Spanner instance or
+database name. See [Options](#options).
+
+**WARNING: Please check that permissions for the Spanner instance used by
+HarbourBridge are appropriate. Spanner manages access control at the database
+level, and the database created by HarbourBridge will inherit default
+permissions from the instance. All data written by HarbourBridge is visible to
+anyone who can access the created database.**
+
+As it processes the data, HarbourBridge reports on progress, provides
+stats on the schema and data conversion steps, and an overall assessment of the
+quality of the conversion. It also generates a schema file, report file and
+a session file (and a bad-data file if data was dropped). See
+[Files Generated by HarbourBridge](#files-generated-by-harbourbridge). Details
+of how source database's schema is mapped to Spanner can be found in the
+[Schema Conversion](#schema-conversion) section.
+
+This tool is part of the Cloud Spanner Ecosystem, a community contributed and
+supported open source repository. Please [report
+issues](https://github.com/cloudspannerecosystem/harbourbridge/issues) and send
+pull requests. See the [HarbourBridge
+Whitepaper](https://github.com/cloudspannerecosystem/harbourbridge/blob/master/whitepaper.md)
+for a discussion of our plans for the tool.
+
+Note that the HarbourBridge tool is not an officially supported Google product
+and is not officially supported as part of the Cloud Spanner product.
+
+## Quickstart Guide
+
+### Before you begin
+
+Complete the steps described in
+[Set up](https://cloud.google.com/spanner/docs/getting-started/set-up), which
+covers creating and setting a default Google Cloud project, enabling billing,
+enabling the Cloud Spanner API, and setting up OAuth 2.0 to get authentication
+credentials to use the Cloud Spanner API.
+
+In particular, ensure that you run
+
+```sh
+gcloud auth application-default login
+```
+
+to set up your local development environment with authentication credentials.
+
+Set the GCLOUD_PROJECT environment variable to your Google Cloud project ID:
+
+```sh
+export GCLOUD_PROJECT=my-project-id
+```
+
+If you do not already have a Cloud Spanner instance, or you want to use a
+separate instance specifically for running HarbourBridge, then create a Cloud
+Spanner instance by following the "Create an instance" instructions on the
+[Quickstart using the console](https://cloud.google.com/spanner/docs/quickstart-console)
+guide. HarbourBridge will create a database for you, but it will not create a
+Spanner instance.
+
+Install Go ([download](https://golang.org/doc/install)) on your development
+machine if it is not already installed, configure the GOPATH environment
+variable if it is not already configured, and
+[test your installation](https://golang.org/doc/install#testing).
+
+### Installing HarbourBridge
+
+Download the tool to your machine and install it.
+
+```sh
+GO111MODULE=on go get github.com/cloudspannerecosystem/harbourbridge
+```
+
+The tool should now be installed as `$GOPATH/bin/harbourbridge`.
+See the [Troubleshooting Guide](#troubleshooting-guide) if you run
+into any issues.
+
+### Running HarbourBridge
+
+To use the tool on a PostgreSQL database called mydb, run
+
+```sh
+pg_dump mydb > mydb.pg_dump
+$GOPATH/bin/harbourbridge schema -source=postgresql < mydb.pg_dump
+```
+
+To use the tool on a MySQL database called mydb, run
+
+```sh
+mysqldump mydb > mydb.mysqldump
+$GOPATH/bin/harbourbridge schema -source=mysql < mydb.mysqldump
+```
+
+To use the tool on a DynamoDB database, run
+
+```sh
+$GOPATH/bin/harbourbridge schema -source=dynamodb
+```
+More details on running harbourbridge can be found in [Example usage](#example-usage) section.
+
+This command will use the cloud project specified by the `GCLOUD_PROJECT`
+environment variable, automatically determine the Cloud Spanner instance
+associated with this project, convert the source schema to a Spanner schema 
+(For MySQL/Postgres) or infer a schema from the DynamoDB instance, create a
+new Cloud Spanner database with this schema, and finally, populate this new
+database with the data from the source database. 
+If the project has multiple instances, then list of available instances 
+will be shown and you will have to pick one of the available instances and 
+set the `--instance` flag in target-profile. The new Cloud Spanner database
+will have a name of the form `{SOURCE}_{DATE}_{RANDOM}`, where`{SOURCE}` is
+the source specified,`{DATE}`is today's date, and`{RANDOM}` is a random
+suffix for uniqueness.
+
+See the [Troubleshooting Guide](#troubleshooting-guide) for help on debugging
+issues.
+
+HarbourBridge also [generates several files](#files-generated-by-harbourbridge)
+when it runs: a schema file, a report file (with detailed analysis of the
+conversion), a session file and a bad data file (if any data was dropped).
+
+### Sample Dump Files
+
+If you don't have ready access to a PostgreSQL or MySQL database, some example
+dump files can be found [here](examples). The files
+[cart.pg_dump](examples/cart.pg_dump) and
+[cart.mysqldump](examples/cart.mysqldump) contain pg_dump and mysqldump output
+for a very basic shopping cart application (just two tables, one for products
+and one for user carts). The files [singers.pg_dump](examples/singers.pg_dump)
+and [singers.mysqldump](examples/singers.mysqldump) contain pg_dump and
+mysqldump output for a version of the [Cloud Spanner
+singers](https://cloud.google.com/spanner/docs/schema-and-data-model#creating_a_table)
+example. To use HarbourBridge on cart.pg_dump, download the file locally and
+run
+
+```
+$GOPATH/bin/harbourbridge schema -source=postgresql < cart.pg_dump
+```
+
+### Verifying Results
+
+Once the tool has completed, you can verify the new database and its content
+using the Google Cloud Console. Go to the [Cloud Spanner Instances
+page](https://console.cloud.google.com/spanner/instances), select your Spanner
+instance, and then find the database created by HarbourBridge and select
+it. This will list the tables created by HarbourBridge. Select a table, and take
+a look at its schema and data. Next, go to the query page, and try
+some SQL statements. For example
+
+```
+SELECT COUNT(*) from mytable
+```
+
+to check the number of rows in table `mytable`.
+
+### Next Steps
+
+The tables created by HarbourBridge provide a starting point for evaluation of
+Spanner. While they preserve much of the core structure of your PostgreSQL/MySQL
+schema and data, many key features have been dropped, including functions, 
+sequences, procedures,triggers, and views. For DynamoDB, the conversion from 
+schemaless to schema is focused on the use-case where customers use DynamoDB in 
+a consistent, structured way with a fairly well defined set of columns and types.
+
+As a result, the out-of-the-box performance you get from these tables could be
+slower than what you get from PostgreSQL/MySQL/DynamoDB. 
+
+To improve performance, also consider using [Interleaved
+Tables](https://cloud.google.com/spanner/docs/schema-and-data-model#creating-interleaved-tables)
+to tune performance.
+
+View HarbourBridge as a base set of functionality for Spanner evalution that can
+be readily expanded. Consider forking and modifying the codebase to add the
+functionality you need. Please [file
+issues](https://github.com/cloudspannerecosystem/harbourbridge/issues) and send
+PRs for fixes and new functionality. See our backlog of [open
+issues](https://github.com/cloudspannerecosystem/harbourbridge/issues). Our
+plans and aspirations for developing HarbourBridge further are outlined in the
+[HarbourBridge
+Whitepaper](https://github.com/cloudspannerecosystem/harbourbridge/blob/master/whitepaper.md).
+
+You can also change the way HarbourBridge behaves by directly editing the
+pg_dump/mysqldump output. For example, suppose you want to try out different 
+primary keys for a table. First run pg_dump/mysqldump and save the output to 
+a file. Then modify (or add) the relevant 
+`ALTER TABLE ... ADD CONSTRAINT ... PRIMARY KEY ...` statement in the 
+pg_dump/mysqldump output file so that the primary keys match what you need. 
+Then run HarbourBridge on the modified pg_dump/mysqldump output.
+
+## Files Generated by HarbourBridge
+
+HarbourBridge generates several files as it runs:
+
+- Schema file (ending in `schema.txt`): contains the generated Spanner
+  schema, interspersed with comments that cross-reference to the relevant
+  PostgreSQL/MySQL schema definitions.
+
+- Session file (ending in `session.json`): contains all schema and data
+  conversion state endcoded as JSON. It is basically a snapshot of the session.
+
+- Report file (ending in `report.txt`): contains a detailed analysis of the
+  PostgreSQL/MySQL to Spanner migration, including table-by-table stats and an
+  analysis of PostgreSQL/MySQL types that don't cleanly map onto Spanner types. 
+  Note that PostgreSQL/MySQL types that don't have a corresponding Spanner type 
+  are mapped to STRING(MAX).
+
+- Bad data file (ending in `dropped.txt`): contains details of data
+  that could not be converted and written to Spanner, including sample
+  bad-data rows. If there is no bad-data, this file is not written (and we
+  delete any existing file with the same name from a previous run).
+
+By default, these files are prefixed by the name of the Spanner database (with a
+dot separator). The file prefix can be overridden using the `-prefix`
+[option](#options).
+
+## Command line flags
+
+`-source` Required flag. Specifies the source database. Supported source
+databases are _'postgres'_, _'mysql'_ and _'dynamodb'_.
+
+`-target` Optional flag. Specifies the target database. Defaults to _'spanner'_ 
+, which is the only supported target database today.
+
+`-prefix` Specifies a file prefix for the report, schema, and bad-data files
+written by the tool. If no file prefix is specified, the name of the Spanner
+database (plus a '.') is used.
+
+`-v` or `-verbose` Specifies verbose mode. This will cause HarbourBridge to
+output detailed messages about the conversion.
+
+`-skip-foreign-keys` Controls whether we add foreign key constraints after
+data migration is complete. This flag cannot be used with schema-only mode,
+and does not affect the generation of foreign key statements during schema
+processing i.e. foreign key constraints will still appear in the generated 
+Spanner DDL files.
+
+`-session` Specifies a session file that contains all schema and data 
+conversion state endcoded as JSON.
+
+### Source profile (`-source-profile`)
+
+HarbourBridge accepts the following options for --source-profile,
+specified as "key1=value1,key2=value,..." pairs:
+
+`-file` Specifies the full path of the file to use for reading source database
+schema and/or data. This can also point to a file stored in GCS e.g., 
+gcs://<bucket_name>/<file_path>. This flag is optional, and file can also be
+piped to stdin, if available locally.
+
+`-format` Specifies the format of the file. This flag is also optional, and
+defaults to `dump`. This may be extended in future to support other formats
+such as `csv`, `avro` etc.
+
+### Target profile (`-target-profile`)
+
+HarbourBridge accepts the following options for --target-profile,
+specified as "key1=value1,key2=value,..." pairs:
+
+`-dbname` Specifies the name of the Spanner database to create. This must be a
+new database. If dbname is not specified, HarbourBridge creates a new unique
+dbname.
+
+`-instance` Specifies the Spanner instance to use. The new database will be
+created in this instance. If not specified, the tool automatically determines an
+appropriate instance using gcloud.
+
+## Example Usage
+
+Details on HarbourBridge example usage can be found here: 
+- [PostgreSQL example usage](sources/postgres/README.md#example-postgresql-usage)
+- [MySQL example usage](sources/mysql/README.md#example-mysql-usage)
+- [DynamoDB example usage](sources/dynamodb/README.md#example-dynamodb-usage)
+
+
+## Schema Conversion
+
+Details on HarbourBridge schema conversion can be found here:
+- [PostgreSQL schema conversion](sources/postgres/README.md#schema-conversion)
+- [MySQL schema conversion](sources/mysql/README.md#schema-conversion)
+- [DynamoDB schema conversion](sources/dynamodb/README.md#schema-conversion)
+
+## Data Conversion
+
+HarbourBridge converts PostgreSQL/MySQL/DynamoDB data to Spanner data based on 
+the Spanner schema it constructs. Conversion for most data types is fairly 
+straightforward, but several types deserve discussion. Note that HarbourBridge 
+is not intended for databases larger than about 100GB. Details on HarbourBridge 
+data conversion can be found here:
+- [PostgreSQL data conversion](sources/postgres/README.md#data-conversion)
+- [MySQL data conversion](sources/mysql/README.md#data-conversion)
+- [DynamoDB data conversion](sources/dynamodb/README.md#data-conversion)
+
+## Troubleshooting Guide
+
+HarbourBridge is written using the Go module system, and so it must be
+installed in [module-aware mode](https://golang.org/cmd/go/#hdr-Module_support).
+This can be achived by setting the environment variable `GO111MODULE` to `on`
+before calling "go get" e.g.
+
+```sh
+GO111MODULE=on go get github.com/cloudspannerecosystem/harbourbridge
+```
+
+As an alternative to "go get", you can make a copy of the HarbourBridge
+codebase from the github repository and use "go run":
+
+```sh
+git clone https://github.com/cloudspannerecosystem/harbourbridge
+cd harbourbridge
+pg_dump mydb > mydb.pg_dump
+go run github.com/cloudspannerecosystem/harbourbridge schema -source=postgresql < mydb.pg_dump
+```
+
+This workflow also allows you to modify or customize the HarbourBridge codebase.
+
+The following steps can help diagnose common issues encountered while running
+HarbourBridge.
+
+### 1. Verify driver configuration
+
+First, check that the driver is correctly configured to connect to your
+database. Driver configuration varies depending on the driver used.
+
+#### 1.1 pg_dump
+
+If you are using pg_dump (-driver=pg_dump), check that pg_dump is
+correctly configured to connect to your PostgreSQL
+database. Note that pg_dump uses the same options as psql to connect to your
+database. See the [psql](https://www.postgresql.org/docs/9.3/app-psql.html) and
+[pg_dump](https://www.postgresql.org/docs/9.3/app-pgdump.html) documentation.
+
+Access to a PostgreSQL database is typically configured using the
+_PGHOST_, _PGPORT_, _PGUSER_, _PGDATABASE_ environment variables,
+which are standard across PostgreSQL utilities.
+
+It is also possible to configure access via pg_dump's command-line options
+`--host`, `--port`, and `--username`.
+
+#### 1.2 Direct access to PostgreSQL
+
+In this case, HarbourBridge connects directly to the PostgreSQL
+database to retrieve table schema and data. Set the `-driver=postgres`
+and provide the environment variables `PGHOST`, `PGPORT`, `PGUSER`, 
+`PGDATABASE`. Password can be specified either in the `PGPASSWORD`
+environment variable or provided at the password prompt.
+
+#### 1.3 mysqldump
+
+If you are using mysqldump (-driver=mysqldump), check that mysqldump is
+correctly configured to connect to your MySQL
+database via the command-line options `--host`, `--port`, and `--user`.
+Note that mysqldump uses the same options as mysql to connect to your
+database. See the [mysql](https://dev.mysql.com/doc/refman/8.0/en/mysql-commands.html)
+and [mysqldump](https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html) documentation.
+
+#### 1.4 Direct access to MySQL
+
+In this case, HarbourBridge connects directly to the MySQL
+database to retrieve table schema and data. Set the `-driver=mysql`
+and provide the environment variables `MYSQLHOST`, `MYSQLPORT`, 
+`MYSQLUSER`, `MYSQLDATABASE`. Password can be specified either 
+in the `MYSQLPWD` environment variable or provided at the password 
+prompt.
+
+
+#### 1.5 Direct access to DynamoDB
+
+In this case, HarbourBridge connects directly to the DynamoDB
+database to retrieve table schema and data. Set the `-driver=dynamodb`
+and provide the environment variables `AWS_ACCESS_KEY_ID`, 
+`AWS_SECRET_ACCESS_KEY`, `AWS_REGION`. If you use a custom endpoint for 
+dynamodb, you can specify that using the environment variable 
+`DYNAMODB_ENDPOINT_OVERRIDE`.
+
+### 2. Verify dump output
+
+Next, verify that pg_dump/mysqldump is generating plain-text output. If your 
+database is small, try running
+
+```sh
+{ pg_dump/mysqldump } > file
+```
+
+and look at the output file. It should be a plain-text file containing SQL
+commands. If your database is large, consider just dumping the schema via the
+`--schema-only` for pg_dump and `--no-data` for mysqldump command-line option.
+
+pg_dump/mysqldump can export data in a variety of formats, but HarbourBridge 
+only accepts `plain` format (aka plain-text). See the
+[pg_dump documentation](https://www.postgresql.org/docs/9.3/app-pgdump.html) and
+[mysqldump documentation](https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html) 
+for details about formats.
+
+### 3. Debugging HarbourBridge
+
+The HarbourBridge tool can fail for a number of reasons.
+
+#### 3.1 No space left on device
+
+HarbourBridge needs to read the pg_dump/mysqldump output twice, once to build 
+a schema and once for data ingestion. When pg_dump/mysqldump output is directly
+piped to HarbourBridge, `stdin` is not seekable, and so we write the output to 
+a temporary file. That temporary file is created via Go's ioutil.TempFile. 
+On many systems, this creates a file in `/tmp`, which is sometimes configured 
+with minimal space. A simple workaround is to separately run pg_dump/mysqldump 
+and write its output to a file in a directory with sufficient space. For example,
+if the current working directory has space, then:
+
+```sh
+{ pg_dump/mysqldump } > tmpfile
+harbourbridge < tmpfile
+```
+
+Make sure you cleanup the tmpfile after HarbourBridge has been run. Another
+option is to set the location of Go's TempFile e.g. by setting the `TMPDIR`
+environment variable.
+
+#### 3.2 Unparsable dump output
+
+HarbourBridge uses the [pg_query_go](https://github.com/pganalyze/pg_query_go)
+library for parsing pg_dump and [pingcap parser](https://github.com/pingcap/parser) 
+for parsing mysqldump. It is possible that the pg_dump/mysqldump output is 
+corrupted or uses features that aren't parseable. Parsing errors should 
+generate an error message of the form `Error parsing last 54321 line(s) of input`.
+
+#### 3.2 Credentials problems
+
+HarbourBridge uses standard Google Cloud credential mechanisms for accessing
+Cloud Spanner. If this is mis-configured, you may see errors containing
+"unauthenticated", or "cannot fetch token", or "could not find default
+credentials". You might need to run `gcloud auth application-default login`. 
+See the [Before you begin](#before-you-begin) section for details.
+
+#### 3.4 Can't create database
+
+In this case, the error message printed by the tool should help identify the
+cause. It could be an API permissions issue. For example, the Cloud Spanner API
+may not be appropriately configured. See [Before you begin](#before-you-begin)
+section for details. Alternatively, you have have hit the limit on the number of
+databases per instances (currently 100). This can occur if you re-run the
+HarbourBridge tool many times, since each run creates a new database. In this
+case you'll need to [delete some
+databases](https://cloud.google.com/spanner/docs/getting-started/go/#delete_the_database).
+
+### 4. Database-Specific Issues
+
+The schema, report, and bad-data files [generated by
+HarbourBridge](#files-generated-by-harbourbridge) contain detailed information
+about the schema and data conversion process, including issues and problems
+encountered.
+
+### 5. Reporting Issues
+
+If you are having problems with HarbourBridge, please [submit an
+issue](https://github.com/cloudspannerecosystem/harbourbridge/issues).
diff --git a/cmd/README.mysql.md b/cmd/README.mysql.md
new file mode 100644
index 0000000..5e9394c
--- /dev/null
+++ b/cmd/README.mysql.md
@@ -0,0 +1,268 @@
+# HarbourBridge: Turnkey MySQL-to-Spanner Evaluation
+
+HarbourBridge is a stand-alone open source tool for Cloud Spanner evaluation,
+using data from an existing PostgreSQL or MySQL database. This README provides
+details of the tool's MySQL capabilities. For general HarbourBridge information
+see this [README](https://github.com/cloudspannerecosystem/harbourbridge#harbourbridge-turnkey-spanner-evaluation).
+
+## Example MySQL Usage
+
+The following examples assume `harbourbridge` has been added to your PATH
+environment variable.
+
+HarbourBridge can either be used with mysqldump or it can be run directly
+on a MySQL database (via go's database/sql package).
+
+### Using HarbourBridge with mysqldump
+
+To use HarbourBridge on a MySQL database called mydb using mysqldump, run:
+
+```sh
+mysqldump mydb | harbourbridge schema -source=mysql
+```
+
+The tool can also be applied to an existing mysqldump file:
+
+```sh
+harbourbridge schema -source=mysql < my_mysqldump_file
+```
+
+To specify a particular Spanner instance to use during data migration, run:
+
+```sh
+harbourbridge data -source=mysql -target-profile="instance=my-spanner-instance" < my_mysqldump_file
+```
+
+By default, HarbourBridge will generate a new Spanner database name to populate.
+You can override this and specify the database name to use by:
+
+```sh
+harbourbridge data -source=mysql -target-profile="dbname=my-spanner-database-name" < my_mysqldump_file
+```
+
+HarbourBridge generates a report file, a schema file, and a bad-data file (if
+there are bad-data rows). You can control where these files are written by
+specifying a file prefix. For example,
+
+```sh
+harbourbridge schema -prefix=mydb. -source=mysql < my_mysqldump_file
+```
+
+will write files `mydb.report.txt`, `mydb.schema.txt`, and
+`mydb.dropped.txt`. The prefix can also be a directory. For example,
+
+```sh
+harbourbridge schema -prefix=~/spanner-eval-mydb/ -source=mysql < my_mysqldump_file
+```
+
+would write the files into the directory `~/spanner-eval-mydb/`. Note
+that HarbourBridge will not create directories as it writes these files.
+
+### Directly connecting to a MySQL database
+
+To use the tool directly on a MySQL database called mydb, run
+
+```sh
+harbourbridge schema -source=mysql
+```
+
+It is assumed that _MYSQLHOST_, _MYSQLPORT_, _MYSQLUSER_, _MYSQLDATABASE_ environment
+variables are set. Password can be specified either in the _MYSQLPWD_ environment
+variable or provided at the password prompt.
+
+Note that all of the options described in the previous section on read from a
+mysqldump file can also be used when directly connecting to a mysql database.
+
+## Schema Conversion
+
+The HarbourBridge tool maps MySQL types to Spanner types as follows:
+
+| MySQL Type                                        | Spanner Type    | Notes                           |
+| ------------------------------------------------- | --------------- | ------------------------------- |
+| `BOOL`, `BOOLEAN`,<br/>`TINYINT(1)`               | `BOOL`          |                                 |
+| `BIGINT`                                          | `INT64`         |                                 |
+| `BINARY`, `VARBINARY`                             | `BYTES(MAX)`    |                                 |
+| `BLOB`, `MEDIUMBLOB`,<br/>`TINYBLOB`, `LONGBLOB`  | `BYTES(MAX)`    |                                 |
+| `BIT`                                             | `BYTES(MAX)`    |                                 |
+| `CHAR`                                            | `STRING(1)`     | CHAR defaults to length 1       |
+| `CHAR(N)`                                         | `STRING(N)`     | c                               |
+| `DATE`                                            | `DATE`          |                                 |
+| `DATETIME`                                        | `TIMESTAMP`     | t                               |
+| `DECIMAL`, `NUMERIC`                              | `NUMERIC`       | p                               |
+| `DOUBLE`                                          | `FLOAT64`       |                                 |
+| `ENUM`                                            | `STRING(MAX)`   |                                 |
+| `FLOAT`                                           | `FLOAT64`       | s                               |
+| `INTEGER`, `MEDIUMINT`,<br/>`TINYINT`, `SMALLINT` | `INT64`         | s                               |
+| `JSON`                                            | `STRING(MAX)`   |                                 |
+| `SET`                                             | `ARRAY<STRING>` | SET only supports string values |
+| `TEXT`, `MEDIUMTEXT`,<br/>`TINYTEXT`, `LONGTEXT`  | `STRING(MAX)`   |                                 |
+| `TIMESTAMP`                                       | `TIMESTAMP`     |                                 |
+| `VARCHAR`                                         | `STRING(MAX)`   |                                 |
+| `VARCHAR(N)`                                      | `STRING(N)`     | c                               |
+
+Spanner does not support `spatial` datatypes of MySQL. Along with `spatial`
+datatypes, all other types map to `STRING(MAX)`. Some of the mappings in this
+table represent potential changes of precision (marked p), differences in
+treatment of timezones (marked t), differences in treatment of fixed-length
+character types (marked c), and changes in storage size (marked s). We discuss
+these, as well as other limits and notes on schema conversion, in the following
+sections.
+
+### `DECIMAL` and `NUMERIC`
+
+[Spanner's NUMERIC
+type](https://cloud.google.com/spanner/docs/data-types#decimal_type) can store
+up to 29 digits before the decimal point and up to 9 after the decimal point.
+MySQL's NUMERIC type can potentially support higher precision than this, so
+please verify that Spanner's NUMERIC support meets your application needs.  Note
+that in MySQL, NUMERIC is implemented as DECIMAL, so the remarks about DECIMAL
+apply equally to NUMERIC.
+
+### `TIMESTAMP` and `DATETIME`
+
+MySQL has two timestamp types: `TIMESTAMP` and `DATETIME`. Both provide
+microsecond resolution, but neither actually stores a timezone with the data.
+The key difference between the two types is that MySQL converts `TIMESTAMP` values
+from the current time zone to UTC for storage, and back from UTC to the current time
+zone for retrieval. This does not occur for `DATETIME` and data is returned without a
+timezone. For `TIMESTAMP`, timezone can be set by time zone offset parameter.
+
+Spanner has a single timestamp type. Data is stored as UTC (there is no separate
+timezone) Spanner client libraries convert timestamps to UTC before sending them
+to Spanner. Data is always returned as UTC. Spanner's timestamp type is
+essentially the same as `TIMESTAMP`, except that there is no analog of
+MySQL's timezone offset parameter.
+
+In other words, mapping MySQL `DATETIME` to `TIMESTAMP` is fairly
+straightforward, but care should be taken with MySQL `DATETIME` data
+because Spanner clients will not drop the timezone.
+
+### `CHAR(n)` and `VARCHAR(n)`
+
+The semantics of fixed-length character types differ between MySQL and
+Spanner. The `CHAR(n)` type in MySQL is right-padded with spaces. If a string
+value smaller than the limit is stored, spaces will be added to pad it out to
+the specified length. If a string longer than the specified length is stored,
+and the extra characters are all spaces, then it will be silently
+truncated. Moreover, trailing spaces are ignored when comparing two values. In
+constrast, Spanner does not give special treatment to spaces, and the specified
+length simply represents the maximum length that can be stored. This is close to
+the semantics of MySQL's `VARCHAR(n)`. However there are some minor
+differences. For example, even `VARCHAR(n)` has some special treatment of
+spaces: string with trailing spaces in excess of the column length are truncated
+prior to insertion and a warning is generated.
+
+### `SET`
+
+MySQL `SET` is a string object that can hold muliple values, each of which must be
+chosen from a list of permitted values specified when the table is created. `SET`
+is being mapped to Spanner type `ARRAY<STRING>`. Validation of `SET` element values
+will be dropped in Spanner. Thus for production use, validation needs to be done
+in the application.
+
+### `Spatial datatype`
+
+MySQL spatial datatypes are used to represent geographic feature.
+It includes `GEOMETRY`, `POINT`, `LINESTRING`, `POLYGON`, `MULTIPOINT`, `MULTIPOLYGON`
+and `GEOMETRYCOLLECTION` datatypes. Spanner does not support spatial data types.
+This datatype are currently mapped to standard `STRING` Spanner datatype.
+
+### Storage Use
+
+The tool maps several MySQL types to Spanner types that use more storage.
+For example, `SMALLINT` is a two-byte integer, but it maps to Spanner's `INT64`,
+an eight-byte integer.
+
+### Primary Keys
+
+Spanner requires primary keys for all tables. MySQL recommends the use of
+primary keys for all tables, but does not enforce this. When converting a table
+without a primary key, HarbourBridge will create a new primary key of type
+INT64. By default, the name of the new column is `synth_id`. If there is already
+a column with that name, then a variation is used to avoid collisions.
+
+### NOT NULL Constraints
+
+The tool preserves `NOT NULL` constraints. Note that Spanner does not require
+primary key columns to be `NOT NULL`. However, in MySQL, a primary key is a
+combination of `NOT NULL` and `UNIQUE`, and so primary key columns from
+MySQL will be mapped to Spanner columns that are both primary keys and `NOT NULL`.
+
+### Foreign Keys
+
+The tool maps MySQL foreign key constraints into Spanner foreign key constraints, and
+preserves constraint names where possible. Since Spanner doesn't support `ON DELETE`
+and `ON UPDATE` actions, we drop them.
+
+### Default Values
+
+Spanner does not currently support default values. We drop these
+MySQL features during conversion.
+
+### Secondary Indexes
+
+The tool maps MySQL secondary indexes to Spanner secondary indexes, and preserves
+constraint names where possible. Note that Spanner requires index key constraint
+names to be globally unique (within a database), but in MySQL they only have to be
+unique for a table, so we add a uniqueness suffix to a name if needed. The tool also
+maps `UNIQUE` constraint into `UNIQUE` secondary index. Note that due to limitations of our
+mysqldump parser, we are not able to handle key column ordering (i.e. ASC/DESC) in
+mysqldump files. All key columns in mysqldump files will be treated as ASC.
+
+### Other MySQL features
+
+MySQL has many other features we haven't discussed, including functions,
+sequences, procedures, triggers, (non-primary) indexes and views. The tool does
+not support these and the relevant statements are dropped during schema
+conversion.
+
+See
+[Migrating from MySQL to Cloud Spanner](https://cloud.google.com/solutions/migrating-mysql-to-spanner)
+for a general discussion of MySQL to Spanner migration issues.
+HarbourBridge follows most of the recommendations in that guide. The main
+difference is that we map a few more types to `STRING(MAX)`.
+
+## Data Conversion
+
+### Timestamps and Timezones
+
+As noted earlier when discussing [schema conversion of
+TIMESTAMP](#timestamp), there are some subtle differences in how timestamps are
+handled in MySQL and Spanner.
+
+During data conversion, MySQL `TIMESTAMP` values are converted to UTC and
+stored in Spanner. The conversion proceeds as follows. If the value has a
+timezone offset, that timezone is respected during the conversion to UTC. If the value
+does not have a timezone offset, then we look for any `set timezone` statements in the
+mysqldump output and use the timezone offset specified. Otherwise, we use '+00:00' timezone offset (UTC).
+
+### Strings, character set support and UTF-8
+
+Spanner requires that `STRING` values be UTF-8 encoded. All Spanner functions
+and operators that act on `STRING` values operate on Unicode characters rather
+than bytes. Since we map many MySQL types (including `TEXT` and `CHAR`
+types) to Spanner's `STRING` type, HarbourBridge is effectively a UTF-8 based
+tool.
+
+Note that the tool itself does not do any encoding/decoding or UTF-8 checks: it
+passes through data from mysqldump to Spanner. Internally, we use Go's string
+type, which supports UTF-8.
+
+### Spatial datatypes support
+
+As noted earlier when discussing [schema conversion of
+Spatial datatype](#spatial-datatype), Spanner does not support spatial datatypes and are
+mapped to `STRING(MAX)` Spanner type. Data conversion for spatial datatypes is different
+for both `mysql` and `mysqldump` driver.
+
+- MySQL information schema approach (`-driver=mysql`) : Data from MySQL is fetched using
+  'ST_AsText(g)' function which converts a value in internal geometry format to its WKT(Well-Known Text)
+  representation and returns the string result. This value will be stored as `STRING` in Spanner.
+- MySQL dump approach (`-driver=mysqldump`) : Mysqldump will have the internal geometry data in
+  binary format. It cannot be converted to WKT format and there is no proper method for mysqldump
+  generation of spatial datatypes also. Thus, this value will just be fetched as a `TEXT` type and
+  converted to Spanner type `STRING`.
+
+Note that mysql information schema approach would be the recommended approach for data conversion of
+spatial datatypes. For production use, you must store this data using standard data types, and implement
+any searching/filtering logic in the application layer.
diff --git a/cmd/README.postgres.md b/cmd/README.postgres.md
new file mode 100644
index 0000000..ab78960
--- /dev/null
+++ b/cmd/README.postgres.md
@@ -0,0 +1,257 @@
+# HarbourBridge: Turnkey PostgreSQL-to-Spanner Evaluation
+
+HarbourBridge is a stand-alone open source tool for Cloud Spanner evaluation,
+using data from an existing PostgreSQL or MySQL database. This README provides
+details of the tool's PostgreSQL capabilities. For general HarbourBridge information
+see this [README](https://github.com/cloudspannerecosystem/harbourbridge#harbourbridge-turnkey-spanner-evaluation).
+
+## Example PostgreSQL Usage
+
+The following examples assume `harbourbridge` has been added to your PATH
+environment variable.
+
+HarbourBridge can either be used with pg_dump or it can be run directly
+on a PostgreSQL database (via go's database/sql package).
+
+### Using HarbourBridge with pg_dump
+
+To use HarbourBridge on a PostgreSQL database called mydb using pg_dump output,run:
+
+```sh
+pg_dump mydb | harbourbridge schema -source=postgresql
+```
+
+You can use one of `postgresql`, `postgres`, or `pg` to specify PostgreSQL as
+the source database.
+
+The tool can also be applied to an existing pg_dump file:
+
+```sh
+harbourbridge schema -source=postgres < my_pg_dump_file
+```
+
+To specify a particular Spanner instance to use during data migration, run:
+
+```sh
+harbourbridge data -source=postgres -target-profile="instance=my-spanner-instance" < my_pg_dump_file
+```
+
+By default, HarbourBridge will generate a new Spanner database name to populate.
+You can override this and specify the database name to use by:
+
+```sh
+harbourbridge data -source=postgres -target-profile="dbname=my-spanner-database-name,instance=my-spanner-instance" < my_pg_dump_file
+```
+
+HarbourBridge generates a report file, a schema file, and a bad-data file (if
+there are bad-data rows during data migration). You can control where these
+files are written by specifying a file prefix. For example,
+
+```sh
+harbourbridge schema -prefix=mydb. -source=pg < my_pg_dump_file
+```
+
+will write files `mydb.report.txt`, `mydb.schema.txt`, and
+`mydb.dropped.txt`. The prefix can also be a directory. For example,
+
+```sh
+harbourbridge schema -prefix=~/spanner-eval-mydb/ -source=pg < my_pg_dump_file
+```
+
+would write the files into the directory `~/spanner-eval-mydb/`. Note
+that HarbourBridge will not create directories as it writes these files.
+
+### Directly connecting to a PostgreSQL database
+
+To use the tool directly on a PostgresSQL database called mydb, run
+
+```sh
+harbourbridge schema -source=postgres
+```
+
+It is assumed that _PGHOST_, _PGPORT_, _PGUSER_, _PGDATABASE_ environment
+variables are set. Password can be specified either in the _PGPASSWORD_
+environment variable or provided at the password prompt.
+
+Note that all of the options described in the previous section when specifying
+a dump file can also be used when directly connecting to a PostgreSQL database.
+
+## Schema Conversion
+
+The HarbourBridge tool maps PostgreSQL types to Spanner types as follows:
+
+| PostgreSQL Type    | Spanner Type           | Notes                                     |
+| ------------------ | ---------------------- | ----------------------------------------- |
+| `BOOL`             | `BOOL`                 |                                           |
+| `BIGINT`           | `INT64`                |                                           |
+| `BIGSERIAL`        | `INT64`                | a                                         |
+| `BYTEA`            | `BYTES(MAX)`           |                                           |
+| `CHAR`             | `STRING(1)`            | CHAR defaults to length 1                 |
+| `CHAR(N)`          | `STRING(N)`            | c                                         |
+| `DATE`             | `DATE`                 |                                           |
+| `DOUBLE PRECISION` | `FLOAT64`              |                                           |
+| `INTEGER`          | `INT64`                | s                                         |
+| `NUMERIC`          | `NUMERIC`              | p                                         |
+| `REAL`             | `FLOAT64`              | s                                         |
+| `SERIAL`           | `INT64`                | a, s                                      |
+| `SMALLINT`         | `INT64`                | s                                         |
+| `TEXT`             | `STRING(MAX)`          |                                           |
+| `TIMESTAMP`        | `TIMESTAMP`            | t                                         |
+| `TIMESTAMPTZ`      | `TIMESTAMP`            |                                           |
+| `VARCHAR`          | `STRING(MAX)`          |                                           |
+| `VARCHAR(N)`       | `STRING(N)`            | c                                         |
+| `ARRAY(`pgtype`)`  | `ARRAY(`spannertype`)` | if scalar type pgtype maps to spannertype |
+
+All other types map to `STRING(MAX)`. Some of the mappings in this table
+represent potential changes of precision (marked p), dropped autoincrement
+functionality (marked a), differences in treatment of timezones (marked t),
+differences in treatment of fixed-length character types (marked c), and changes
+in storage size (marked s). We discuss these, as well as other limits and notes
+on schema conversion, in the following sections.
+
+### `NUMERIC`
+
+[Spanner's NUMERIC
+type](https://cloud.google.com/spanner/docs/data-types#decimal_type) can store
+up to 29 digits before the decimal point and up to 9 after the decimal point.
+PostgreSQL's NUMERIC type can potentially support higher precision that this, so
+please verify that Spanner's NUMERIC support meets your application needs.
+
+### `BIGSERIAL` and `SERIAL`
+
+Spanner does not support autoincrementing types, so these both map to `INT64`
+and the autoincrementing functionality is dropped.
+
+### `TIMESTAMP`
+
+PosgreSQL has two timestamp types: `TIMESTAMP` and `TIMESTAMPTZ`. Both have an 8
+byte data representation and provide microsecond resolution, but neither
+actually stores a timezone with the data. The key difference between the two
+types is how string literals are converted to timestamps and queries return
+data. For `TIMESTAMP`, all timezone information is dropped, and data is returned
+without a timezone. For `TIMESTAMPTZ`, string literals are converted to UTC,
+using the literal's timezone if it is specified, or the PostgreSQL's timezone
+paramater if not. When data is printed stored data (in UTC) is converted to the
+timezone from the timezone parameter
+
+Spanner has a single timestamp type. Data is stored as UTC (there is no separate
+timezone) Spanner client libraries convert timestamps to UTC before sending them
+to Spanner. Data is always returned as UTC. Spanner's timestamp type is
+essentially the same as `TIMESTAMPTZ`, except that there is no analog of
+PostgreSQL's timezone parameter.
+
+In other words, mapping PostgreSQL `TIMESTAMPTZ` to `TIMESTAMP` is fairly
+straightforward, but care should be taken with PostgreSQL `TIMESTAMP` data
+because Spanner clients will not drop the timezone.
+
+### `CHAR(n)` and `VARCHAR(n)`
+
+The semantics of fixed-length character types differ between PostgreSQL and
+Spanner. The `CHAR(n)` type in PostgreSQL is padded with spaces. If a string
+value smaller than the limit is stored, spaces will be added to pad it out to
+the specified length. If a string longer than the specified length is stored,
+and the extra characters are all spaces, then it will be silently
+truncated. Moreover, trailing spaces are ignored when comparing two values. In
+constrast, Spanner does not give special treatment to spaces, and the specified
+length simply represents the maximum length that can be stored. This is close to
+the semantics of PostgreSQL's `VARCHAR(n)`. However there are some minor
+differences. For example, even `VARCHAR(n)` has some special treatment of
+spaces: strings longer than the specified length are silently truncated if the
+extra characters are all spaces.
+
+### Storage Use
+
+The tool maps several PostgreSQL types to Spanner types that use more storage.
+For example, `SMALLINT` is a two-byte integer, but it maps to Spanner's `INT64`,
+an eight-byte integer. This additional storage could be significant for large
+arrays.
+
+### Arrays
+
+Spanner does not support multi-dimensional arrays. So while `TEXT[4]` maps to
+`ARRAY<STRING(MAX)>` and `REAL ARRAY` maps to `ARRAY<FLOAT64>`, `TEXT[][]` maps
+to `STRING(MAX)`.
+
+Also note that PosgreSQL supports array limits, but the PostgreSQL
+implementation ignores them. Spanner does not support array size limits, but
+since they have no effect anyway, the tool just drops them.
+
+### Primary Keys
+
+Spanner requires primary keys for all tables. PostgreSQL recommends the use of
+primary keys for all tables, but does not enforce this. When converting a table
+without a primary key, HarbourBridge will create a new primary key of type
+INT64. By default, the name of the new column is `synth_id`. If there is already
+a column with that name, then a variation is used to avoid collisions.
+
+### NOT NULL Constraints
+
+The tool preserves `NOT NULL` constraints. Note that Spanner does not require
+primary key columns to be `NOT NULL`. However, in PostgreSQL, a primary key is a
+combination of `NOT NULL` and `UNIQUE`, and so primary key columns from
+PostgreSQL will be mapped to Spanner columns that are both primary keys and `NOT NULL`.
+
+### Foreign Keys
+
+The tool maps PostgreSQL foreign key constraints into Spanner foreign key constraints, and
+preserves constraint names where possible. Note that Spanner requires foreign key
+constraint names to be globally unique (within a database), but in postgres they only
+have to be unique for a table, so we add a uniqueness suffix to a name if needed.
+Spanner doesn't support `ON DELETE` and `ON UPDATE` actions, so we drop these.
+
+### Default Values
+
+Spanner does not currently support default values. We drop these
+PostgreSQL features during conversion.
+
+### Secondary Indexes
+
+The tool maps PostgresSQL secondary indexes to Spanner secondary indexes, preserving
+constraint names where possible. The tool also maps PostgreSQL `UNIQUE` constraints to
+Spanner `UNIQUE` secondary indexes. Check [here](https://cloud.google.com/spanner/docs/migrating-postgres-spanner#indexes)
+for more details.
+
+### Other PostgreSQL features
+
+PostgreSQL has many other features we haven't discussed, including functions,
+sequences, procedures, triggers, (non-primary) indexes and views. The tool does
+not support these and the relevant statements are dropped during schema
+conversion.
+
+See
+[Migrating from PostgreSQL to Cloud Spanner](https://cloud.google.com/spanner/docs/migrating-postgres-spanner)
+for a general discussion of PostgreSQL to Spanner migration issues.
+HarbourBridge follows most of the recommendations in that guide. The main
+difference is that we map a few more types to `STRING(MAX)`.
+
+## Data Conversion
+
+### Timestamps and Timezones
+
+As noted earlier when discussing [schema conversion of
+TIMESTAMP](#timestamp), there are some subtle differences in how timestamps are
+handled in PostgreSQL and Spanner.
+
+During data conversion, PostgreSQL `TIMESTAMPTZ` values are converted to UTC and
+stored in Spanner. The conversion proceeds as follows. If the value has a
+timezone, that timezone is respected during the conversion to UTC. If the value
+does not have a timezone, then we look for any `set timezone` statements in the
+pg_dump output and use the timezone specified. Otherwise, we use the `TZ`
+environment variable as the timezone, and failing that, we use the local system
+timezone default (as determined by Go).
+
+In constrast, conversion of PostgreSQL `TIMESTAMP` values proceeds by ignoring
+any timezone information and just treating the value as UTC and storing it in
+Spanner.
+
+### Strings, character set support and UTF-8
+
+Spanner requires that `STRING` values be UTF-8 encoded. All Spanner functions
+and operators that act on `STRING` values operate on Unicode characters rather
+than bytes. Since we map many PostgreSQL types (including `TEXT` and `CHAR`
+types) to Spanner's `STRING` type, HarbourBridge is effectively a UTF-8 based
+tool.
+
+Note that the tool itself does not do any encoding/decoding or UTF-8 checks: it
+passes through data from pg_dump to Spanner. Internally, we use Go's string
+type, which supports UTF-8.
diff --git a/cmd/common.go b/cmd/common.go
new file mode 100644
index 0000000..301dd11
--- /dev/null
+++ b/cmd/common.go
@@ -0,0 +1,43 @@
+package cmd
+
+import (
+	"encoding/csv"
+	"fmt"
+	"strings"
+)
+
+// Parses input string `s` as a map of key-value pairs. It's expected that the
+// input string is of the form "key1=value1,key2=value2,..." etc. Return error
+// otherwise.
+func parseProfile(s string) (map[string]string, error) {
+	params := make(map[string]string)
+	if len(s) == 0 {
+		return params, nil
+	}
+
+	// We use CSV reader to parse key=value pairs separated by a comma to
+	// handle the case where a value may contain a comma within a quote. We
+	// expect exactly one record to be returned.
+	r := csv.NewReader(strings.NewReader(s))
+	r.Comma = ','
+	r.TrimLeadingSpace = true
+	records, err := r.ReadAll()
+	if err != nil {
+		return params, err
+	}
+	if len(records) > 1 {
+		return params, fmt.Errorf("contains invalid newline characters")
+	}
+
+	for _, kv := range records[0] {
+		s := strings.Split(strings.TrimSpace(kv), "=")
+		if len(s) != 2 {
+			return params, fmt.Errorf("invalid key=value pair (expected format: key1=value1): %v", kv)
+		}
+		if _, ok := params[s[0]]; ok {
+			return params, fmt.Errorf("duplicate key found: %v", s[0])
+		}
+		params[s[0]] = s[1]
+	}
+	return params, nil
+}
diff --git a/cmd/schema.go b/cmd/schema.go
index 7823fe3..b037477 100644
--- a/cmd/schema.go
+++ b/cmd/schema.go
@@ -4,7 +4,6 @@ import (
 	"context"
 	"flag"
 	"fmt"
-	"log"
 	"os"
 	"path"
 	"time"
@@ -16,11 +15,11 @@ import (
 
 // SchemaCmd struct with flags.
 type SchemaCmd struct {
-	filePrefix       string
-	driverName       string
-	schemaSampleSize int64
-	dumpFilePath     string
-	targetDb         string
+	source        string
+	sourceProfile string
+	target        string
+	targetProfile string
+	filePrefix    string // TODO: move filePrefix to global flags
 }
 
 // Name returns the name of operation.
@@ -35,49 +34,75 @@ func (cmd *SchemaCmd) Synopsis() string {
 
 // Usage returns usage info of the command.
 func (cmd *SchemaCmd) Usage() string {
-	return fmt.Sprintf(`%v schema [file]...
+	return fmt.Sprintf(`%v schema -source=[source] -source-profile="key1=value1,key2=value2" ...
 
 Convert schema for source db specified by driver. Source db dump file can be
-specified by either dump-file flag or piped to stdin. Connection profile for source
-databases in direct connect mode can be specified by setting appropriate
-environment variables. The schema flags are:
+specified by either file param in source-profile or piped to stdin. Connection
+profile for source databases in direct connect mode can be specified by setting
+appropriate environment variables. The schema flags are:
 `, path.Base(os.Args[0]))
 }
 
 // SetFlags sets the flags.
 func (cmd *SchemaCmd) SetFlags(f *flag.FlagSet) {
+	f.StringVar(&cmd.source, "source", "", "Flag for specifying source DB, (e.g., `PostgreSQL`, `MySQL`, `DynamoDB`)")
+	f.StringVar(&cmd.sourceProfile, "source-profile", "", "Flag for specifying connection profile for source database e.g., \"file=<path>,format=dump\"")
+	f.StringVar(&cmd.target, "target", "Spanner", "Specifies the target DB, defaults to Spanner (accepted values: `Spanner`)")
+	f.StringVar(&cmd.targetProfile, "target-profile", "", "Flag for specifying connection profile for target database e.g., \"dialect=postgresql\"")
 	f.StringVar(&cmd.filePrefix, "prefix", "", "File prefix for generated files")
-	f.StringVar(&cmd.driverName, "driver", "pg_dump", "Flag for specifying source DB or dump files (accepted values are \"pg_dump\", \"postgres\", \"mysqldump\", \"mysql\", and \"dynamodb\")")
-	f.Int64Var(&cmd.schemaSampleSize, "schema-sample-size", int64(100000), "Number of rows to use for inferring schema (only for DynamoDB)")
-	f.StringVar(&cmd.dumpFilePath, "dump-file", "", "Location of dump file to process")
-	f.StringVar(&cmd.targetDb, "target-db", "", "Specifies the target DB, defaults to spanner")
 }
 
 func (cmd *SchemaCmd) Execute(ctx context.Context, f *flag.FlagSet, _ ...interface{}) subcommands.ExitStatus {
-	ioHelper := conversion.NewIOStreams(cmd.driverName, cmd.dumpFilePath)
+	sourceProfile, err := NewSourceProfile(cmd.sourceProfile, cmd.source)
+	if err != nil {
+		panic(err)
+	}
+	driverName, err := sourceProfile.ToLegacyDriver(cmd.source)
+	if err != nil {
+		panic(err)
+	}
+
+	targetProfile, err := NewTargetProfile(cmd.targetProfile)
+	if err != nil {
+		panic(err)
+	}
+	targetDb := targetProfile.ToLegacyTargetDb()
+
+	dumpFilePath := ""
+	if sourceProfile.ty == SourceProfileTypeFile && (sourceProfile.file.format == "" || sourceProfile.file.format == "dump") {
+		dumpFilePath = sourceProfile.file.path
+	}
+	ioHelper := conversion.NewIOStreams(driverName, dumpFilePath)
 	if ioHelper.SeekableIn != nil {
 		defer ioHelper.In.Close()
 	}
 
 	// If filePrefix not explicitly set, use generated dbName.
 	if cmd.filePrefix == "" {
-		dbName, err := conversion.GetDatabaseName(cmd.driverName, time.Now())
+		dbName, err := conversion.GetDatabaseName(driverName, time.Now())
 		if err != nil {
-			log.Fatalf("can't generate database name for prefix: %v\n", err)
+			panic(fmt.Errorf("can't generate database name for prefix: %v", err))
 		}
 		cmd.filePrefix = dbName + "."
 	}
 
+	schemaSampleSize := int64(100000)
+	if sourceProfile.ty == SourceProfileTypeConnection {
+		if sourceProfile.conn.ty == SourceProfileConnectionTypeDynamoDB {
+			if sourceProfile.conn.dydb.schemaSampleSize != 0 {
+				schemaSampleSize = sourceProfile.conn.dydb.schemaSampleSize
+			}
+		}
+	}
 	var conv *internal.Conv
-	var err error
-	conv, err = conversion.SchemaConv(cmd.driverName, cmd.targetDb, &ioHelper, cmd.schemaSampleSize)
+	conv, err = conversion.SchemaConv(driverName, targetDb, &ioHelper, schemaSampleSize)
 	if err != nil {
-		log.Fatal(err)
+		panic(err)
 	}
 
 	now := time.Now()
 	conversion.WriteSchemaFile(conv, now, cmd.filePrefix+schemaFile, ioHelper.Out)
 	conversion.WriteSessionFile(conv, cmd.filePrefix+sessionFile, ioHelper.Out)
-	conversion.Report(cmd.driverName, nil, ioHelper.BytesRead, "", conv, cmd.filePrefix+reportFile, ioHelper.Out)
+	conversion.Report(driverName, nil, ioHelper.BytesRead, "", conv, cmd.filePrefix+reportFile, ioHelper.Out)
 	return subcommands.ExitSuccess
 }
diff --git a/cmd/source_profile.go b/cmd/source_profile.go
new file mode 100644
index 0000000..db1ff3e
--- /dev/null
+++ b/cmd/source_profile.go
@@ -0,0 +1,279 @@
+package cmd
+
+import (
+	"fmt"
+	"os"
+	"strconv"
+	"strings"
+)
+
+type SourceProfileType int
+
+const (
+	SourceProfileTypeUnset = iota
+	SourceProfileTypeFile
+	SourceProfileTypeConnection
+	SourceProfileTypeConfig
+)
+
+type SourceProfileFile struct {
+	path   string
+	format string
+}
+
+func NewSourceProfileFile(params map[string]string) SourceProfileFile {
+	profile := SourceProfileFile{}
+	if !filePipedToStdin() {
+		profile.path = params["file"]
+	}
+	if format, ok := params["format"]; ok {
+		profile.format = format
+		// TODO: Add check that format takes values from ["dump", "csv", "avro", ... etc]
+	} else {
+		fmt.Printf("source-profile format defaulting to `dump`\n")
+		profile.format = "dump"
+	}
+	return profile
+}
+
+type SourceProfileConnectionType int
+
+const (
+	SourceProfileConnectionTypeUnset = iota
+	SourceProfileConnectionTypeMySQL
+	SourceProfileConnectionTypePostgreSQL
+	SourceProfileConnectionTypeDynamoDB
+)
+
+type SourceProfileConnectionMySQL struct {
+	host string // Same as MYSQLHOST environment variable
+	port string // Same as MYSQLPORT environment variable
+	user string // Same as MYSQLUSER environment variable
+	db   string // Same as MYSQLDATABASE environment variable
+	pwd  string // Same as MYSQLPWD environment variable
+}
+
+func NewSourceProfileConnectionMySQL(params map[string]string) SourceProfileConnectionMySQL {
+	// TODO: Move parsing of environment variables in this function.
+	mysql := SourceProfileConnectionMySQL{}
+	if host, ok := params["host"]; ok {
+		mysql.host = host
+	}
+	if port, ok := params["port"]; ok {
+		mysql.port = port
+	} else { // Set default port for mysql, which rarely changes.
+		mysql.port = "3306"
+	}
+	if user, ok := params["user"]; ok {
+		mysql.user = user
+	}
+	if db, ok := params["db_name"]; ok {
+		mysql.db = db
+	}
+	if pwd, ok := params["password"]; ok {
+		mysql.pwd = pwd
+	}
+	return mysql
+}
+
+type SourceProfileConnectionPostgreSQL struct {
+	host string // Same as PGHOST environment variable
+	port string // Same as PGPORT environment variable
+	user string // Same as PGUSER environment variable
+	db   string // Same as PGDATABASE environment variable
+	pwd  string // Same as PGPASSWORD environment variable
+}
+
+func NewSourceProfileConnectionPostgreSQL(params map[string]string) SourceProfileConnectionPostgreSQL {
+	pg := SourceProfileConnectionPostgreSQL{}
+	if host, ok := params["host"]; ok {
+		pg.host = host
+	}
+	if port, ok := params["port"]; ok {
+		pg.port = port
+	} else { // Set default port for postgresql, which rarely changes.
+		pg.port = "5432"
+	}
+	if user, ok := params["user"]; ok {
+		pg.user = user
+	}
+	if db, ok := params["db_name"]; ok {
+		pg.db = db
+	}
+	if pwd, ok := params["password"]; ok {
+		pg.pwd = pwd
+	}
+	return pg
+}
+
+type SourceProfileConnectionDynamoDB struct {
+	awsAccessKeyID     string // Same as AWS_ACCESS_KEY_ID environment variable
+	awsSecretAccessKey string // Same as AWS_SECRET_ACCESS_KEY environment variable
+	awsRegion          string // Same as AWS_REGION environment variable
+	dydbEndpoint       string // Same as DYNAMODB_ENDPOINT_OVERRIDE environment variable
+	schemaSampleSize   int64  // Number of rows to use for inferring schema (default 100,000)
+}
+
+func NewSourceProfileConnectionDynamoDB(params map[string]string) (SourceProfileConnectionDynamoDB, error) {
+	// TODO: Populate environment variables for DynamoDB if passed as flags.
+	dydb := SourceProfileConnectionDynamoDB{}
+	if awsAccessKeyID, ok := params["awsAccessKeyID"]; ok {
+		dydb.awsAccessKeyID = awsAccessKeyID
+	}
+	if awsSecretAccessKey, ok := params["awsSecretAccessKey"]; ok {
+		dydb.awsSecretAccessKey = awsSecretAccessKey
+	}
+	if awsRegion, ok := params["awsRegion"]; ok {
+		dydb.awsRegion = awsRegion
+	}
+	if dydbEndpoint, ok := params["dydbEndpoint"]; ok {
+		dydb.dydbEndpoint = dydbEndpoint
+	}
+	if schemaSampleSize, ok := params["schemaSampleSize"]; ok {
+		schemaSampleSizeInt, err := strconv.Atoi(schemaSampleSize)
+		if err != nil {
+			return dydb, fmt.Errorf("could not parse schemaSampleSize = %v as a valid int64", schemaSampleSize)
+		}
+		dydb.schemaSampleSize = int64(schemaSampleSizeInt)
+	}
+	return dydb, nil
+}
+
+type SourceProfileConnection struct {
+	ty    SourceProfileConnectionType
+	mysql SourceProfileConnectionMySQL
+	pg    SourceProfileConnectionPostgreSQL
+	dydb  SourceProfileConnectionDynamoDB
+}
+
+func NewSourceProfileConnection(source string, params map[string]string) (SourceProfileConnection, error) {
+	conn := SourceProfileConnection{}
+	switch strings.ToLower(source) {
+	case "mysql":
+		{
+			conn.ty = SourceProfileConnectionTypeMySQL
+			conn.mysql = NewSourceProfileConnectionMySQL(params)
+		}
+	case "postgresql", "postgres", "pg":
+		{
+			conn.ty = SourceProfileConnectionTypePostgreSQL
+			conn.pg = NewSourceProfileConnectionPostgreSQL(params)
+		}
+	case "dynamodb":
+		{
+			conn.ty = SourceProfileConnectionTypeDynamoDB
+			dydb, err := NewSourceProfileConnectionDynamoDB(params)
+			conn.dydb = dydb
+			if err != nil {
+				return conn, err
+			}
+		}
+	default:
+		return conn, fmt.Errorf("please specify a valid source database using -source flag, received source = %v", source)
+	}
+	return conn, nil
+}
+
+type SourceProfileConfig struct {
+	path string
+}
+
+func NewSourceProfileConfig(path string) SourceProfileConfig {
+	return SourceProfileConfig{path: path}
+}
+
+type SourceProfile struct {
+	ty     SourceProfileType
+	file   SourceProfileFile
+	conn   SourceProfileConnection
+	config SourceProfileConfig
+}
+
+// ToLegacyDriver converts source profile to equivalent legacy global flags
+// e.g., -driver, -dump-file etc since the rest of the codebase still uses the
+// same. TODO: Deprecate this function and pass around SourceProfile across the
+// codebase wherever information about source connection is required.
+func (src SourceProfile) ToLegacyDriver(source string) (string, error) {
+	switch src.ty {
+	case SourceProfileTypeFile:
+		{
+			switch strings.ToLower(source) {
+			case "mysql":
+				return "mysqldump", nil
+			case "postgresql", "postgres", "pg":
+				return "pg_dump", nil
+			case "dynamodb":
+				return "", fmt.Errorf("dump files are not supported with DynamoDB")
+			default:
+				return "", fmt.Errorf("please specify a valid source database using -source flag, received source = %v", source)
+			}
+		}
+	case SourceProfileTypeConnection:
+		{
+			switch strings.ToLower(source) {
+			case "mysql":
+				return "mysql", nil
+			case "postgresql", "postgres", "pg":
+				return "postgres", nil
+			case "dynamodb":
+				return "dynamodb", nil
+			default:
+				return "", fmt.Errorf("please specify a valid source database using -source flag, received source = %v", source)
+			}
+		}
+	case SourceProfileTypeConfig:
+		return "", fmt.Errorf("specifying source-profile using config not implemented")
+	default:
+		return "", fmt.Errorf("invalid source-profile, could not infer type")
+	}
+}
+
+// Source profile is passed as a list of key value pairs on the command line.
+// Following 3 formats are supported as valid source profiles.
+//
+// Format 1. Specify file path and file format.
+// File path can be a local file path or a gcs file path. Support for more file
+// path types can be added in future.
+// File format can be "dump" e.g., when specifying a mysqldump or pgdump etc.
+// Support for more formats e.g., "csv", "avro" etc can be added in future.
+//
+// Example: -source-profile="file=/tmp/abc, format=dump"
+// Example: -source-profile="file=gcs://bucket_name/cart.txt, format=dump"
+//
+// Format 2. Specify source connection parameters. If none specified, then read
+// from envrironment variables.
+//
+// Format 3. Specify a config file that specifies source connection profile.
+//
+func NewSourceProfile(s string, source string) (SourceProfile, error) {
+	if source == "" {
+		return SourceProfile{}, fmt.Errorf("cannot leave -source flag empty, please specify source databases e.g., -source=postgres etc")
+	}
+	params, err := parseProfile(s)
+	if err != nil {
+		return SourceProfile{}, fmt.Errorf("could not parse source profile, error = %v", err)
+	}
+
+	if _, ok := params["file"]; ok || filePipedToStdin() {
+		profile := NewSourceProfileFile(params)
+		return SourceProfile{ty: SourceProfileTypeFile, file: profile}, nil
+	} else if format, ok := params["format"]; ok {
+		// File is not passed in from stdin or specified using "file" flag.
+		return SourceProfile{ty: SourceProfileTypeFile}, fmt.Errorf("file not specified, but format set to %v", format)
+	} else if file, ok := params["config"]; ok {
+		config := NewSourceProfileConfig(file)
+		return SourceProfile{ty: SourceProfileTypeConfig, config: config}, fmt.Errorf("source-profile type config not yet implemented")
+	} else {
+		// Assume connection profile type connection by default, since
+		// connection parameters could be specified as part of environment
+		// variables.
+		conn, err := NewSourceProfileConnection(source, params)
+		return SourceProfile{ty: SourceProfileTypeConnection, conn: conn}, err
+	}
+}
+
+var filePipedToStdin = func() bool {
+	stat, _ := os.Stdin.Stat()
+	// Data is being piped to stdin, if true. Else, stdin is from a terminal.
+	return (stat.Mode() & os.ModeCharDevice) == 0
+}
diff --git a/cmd/target_profile.go b/cmd/target_profile.go
new file mode 100644
index 0000000..ceb954e
--- /dev/null
+++ b/cmd/target_profile.go
@@ -0,0 +1,111 @@
+package cmd
+
+import (
+	"fmt"
+	"strings"
+)
+
+type TargetProfileType int
+
+const (
+	TargetProfileTypeUnset = iota
+	TargetProfileTypeConnection
+)
+
+type TargetProfileConnectionType int
+
+const (
+	TargetProfileConnectionTypeUnset = iota
+	TargetProfileConnectionTypeSpanner
+)
+
+type TargetProfileConnectionSpanner struct {
+	endpoint string // Same as SPANNER_API_ENDPOINT environment variable
+	project  string // Same as GCLOUD_PROJECT environment variable
+	instance string
+	dbname   string
+	dialect  string
+}
+
+type TargetProfileConnection struct {
+	ty TargetProfileConnectionType
+	sp TargetProfileConnectionSpanner
+}
+
+type TargetProfile struct {
+	ty   TargetProfileType
+	conn TargetProfileConnection
+}
+
+// ToLegacyTargetDb converts source profile to equivalent legacy global flag
+// -target-db etc since the rest of the codebase still uses the same.
+// TODO: Deprecate this function and pass around TargetProfile across the
+// codebase wherever information about target connection is required.
+func (trg TargetProfile) ToLegacyTargetDb() string {
+	switch trg.ty {
+	case TargetProfileTypeConnection:
+		{
+			conn := trg.conn
+			switch conn.ty {
+			case TargetProfileConnectionTypeSpanner:
+				{
+					sp := conn.sp
+					if len(sp.dialect) > 0 && strings.ToLower(sp.dialect) == "postgresql" {
+						return "experimental_postgres"
+					}
+					return "spanner"
+				}
+			default:
+				return "spanner"
+			}
+		}
+	default:
+		return "spanner"
+	}
+}
+
+// Target profile is passed as a list of key value pairs on the command line.
+// Today we support only direct connection as a valid target profile type, but
+// in future we can support writing to CSV or AVRO as valid targets.
+//
+// Among direct connection targets, today we only support Spanner database.
+// TargetProfileConnectionType can be extended to add more databases.
+// Users can specify the database dialect, instance, database name etc when
+// connecting to Spanner.
+//
+// Database dialect can take 2 values: GoogleSQL or PostgreSQL and the same
+// correspond to regular Cloud Spanner database and PG Cloud Spanner database
+// respectively.
+//
+// If dbname is not specified, then HarbourBridge will autogenerate the same
+// and create a database with the same name.
+//
+// Example: -target-profile="instance=my-instance1,dbname=my-new-db1"
+// Example: -target-profile="instance=my-instance1,dbname=my-new-db1,dialect=PostgreSQL"
+//
+func NewTargetProfile(s string) (TargetProfile, error) {
+	params, err := parseProfile(s)
+	if err != nil {
+		return TargetProfile{}, fmt.Errorf("could not parse target profile, error = %v", err)
+	}
+
+	sp := TargetProfileConnectionSpanner{}
+	if endpoint, ok := params["endpoint"]; ok {
+		sp.endpoint = endpoint
+	}
+	if project, ok := params["project"]; ok {
+		sp.project = project
+	}
+	if instance, ok := params["instance"]; ok {
+		sp.instance = instance
+	}
+	if dbname, ok := params["dbname"]; ok {
+		sp.dbname = dbname
+	}
+	if dialect, ok := params["dialect"]; ok {
+		sp.dialect = dialect
+	}
+
+	conn := TargetProfileConnection{ty: TargetProfileConnectionTypeSpanner, sp: sp}
+	return TargetProfile{ty: TargetProfileTypeConnection, conn: conn}, nil
+}
diff --git a/main.go b/main.go
index ed9bcea..cd9fab7 100644
--- a/main.go
+++ b/main.go
@@ -100,7 +100,7 @@ func main() {
 	// TODO: Remove this check and always run HB in subcommands mode once
 	// global command line mode is deprecated. We can also enable support for
 	// top-level flags in subcommand then.
-	if os.Args[1] != "" && !strings.HasPrefix(os.Args[1], "-") {
+	if len(os.Args) > 1 && os.Args[1] != "" && !strings.HasPrefix(os.Args[1], "-") {
 		// Using HB CLI in subcommand mode.
 		subcommands.Register(subcommands.HelpCommand(), "")
 		subcommands.Register(subcommands.CommandsCommand(), "")


Test patch:
diff --git a/cmd/source_profile_test.go b/cmd/source_profile_test.go
new file mode 100644
index 0000000..1b0e5dc
--- /dev/null
+++ b/cmd/source_profile_test.go
@@ -0,0 +1,55 @@
+package cmd
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestNewSourceProfileFile(t *testing.T) {
+	testCases := []struct {
+		name         string
+		params       map[string]string
+		pipedToStdin bool
+		want         SourceProfileFile
+	}{
+		{
+			name:         "no params, file piped",
+			params:       map[string]string{},
+			pipedToStdin: true,
+			want:         SourceProfileFile{format: "dump"},
+		},
+		{
+			name:         "format param, file piped",
+			params:       map[string]string{"format": "dump"},
+			pipedToStdin: true,
+			want:         SourceProfileFile{format: "dump"},
+		},
+		{
+			name:         "format and path param, file piped -- piped file takes precedence",
+			params:       map[string]string{"format": "dump", "file": "file1.mysqldump"},
+			pipedToStdin: true,
+			want:         SourceProfileFile{format: "dump"},
+		},
+		{
+			name:         "format and path param, no file piped",
+			params:       map[string]string{"format": "dump", "file": "file1.mysqldump"},
+			pipedToStdin: false,
+			want:         SourceProfileFile{format: "dump", path: "file1.mysqldump"},
+		},
+		{
+			name:         "only path param, no file piped -- default dump format",
+			params:       map[string]string{"file": "file1.mysqldump"},
+			pipedToStdin: false,
+			want:         SourceProfileFile{format: "dump", path: "file1.mysqldump"},
+		},
+	}
+
+	for _, tc := range testCases {
+		// Override filePipedToStdin with the test value.
+		filePipedToStdin = func() bool { return tc.pipedToStdin }
+
+		profile := NewSourceProfileFile(tc.params)
+		assert.Equal(t, profile, tc.want, tc.name)
+	}
+}
diff --git a/testing/postgres/integration_test.go b/testing/postgres/integration_test.go
index 55b67d5..7d10f96 100644
--- a/testing/postgres/integration_test.go
+++ b/testing/postgres/integration_test.go
@@ -178,7 +178,7 @@ func TestIntegration_PGDUMP_SchemaCommand(t *testing.T) {
 	// is because file prefixes use `now` from here (the test function) and
 	// the generated time in the files uses a `now` inside the command, which
 	// can be different.
-	cmd := exec.Command("bash", "-c", fmt.Sprintf("go run github.com/cloudspannerecosystem/harbourbridge schema < %s", dataFilepath))
+	cmd := exec.Command("bash", "-c", fmt.Sprintf("go run github.com/cloudspannerecosystem/harbourbridge schema -source=pg < %s", dataFilepath))
 	var out, stderr bytes.Buffer
 	cmd.Stdout = &out
 	cmd.Stderr = &stderr
@@ -248,7 +248,7 @@ func TestIntegration_POSTGRES_SchemaCommand(t *testing.T) {
 	tmpdir := prepareIntegrationTest(t)
 	defer os.RemoveAll(tmpdir)
 
-	cmd := exec.Command("bash", "-c", fmt.Sprintf("go run github.com/cloudspannerecosystem/harbourbridge schema -driver %s", conversion.POSTGRES))
+	cmd := exec.Command("bash", "-c", "go run github.com/cloudspannerecosystem/harbourbridge schema -source=postgres")
 	var out, stderr bytes.Buffer
 	cmd.Stdout = &out
 	cmd.Stderr = &stderr