Repository: cloudspannerecosystem/harbourbridge
Pull request #79 (2020-08-16 01:43:08)

feat: support DynamoDB schema conversion
This is the first part (schema conversion) of supporting DynamoDB as a source DB. 

## TODOs

* Schema conversion: 
    - [x] initial schema conversion
    - [x] unit tests
    - [x] add `tables` flag
    - [x] add `schema-sample-size` flag
    - [x] convert Number to Numeric if possible; otherwise, convert it to a String

## Idea

The main idea is to scan a certain number of sample data from the table and make a best guess of the type data. If the table is small, we will scan the entire table. 

The algorithm logic can be described as follows: 

1. Calculate the percentage for each data type = count of rows for the data type / count of total scanned rows. 
2. Use an error threshold `0.1%` to remove noise: 
    1. If the percentage is lower than or equal to the error threshold, then we skip this data type. 
    2. If the percentage of Null and NotPresent is lower than or equal to the error threshold, then this column is not nullable (Columns with the Null data type or not presented are considered as the same case). 
3. For the remaining data types (except Null or NotPresent), we use a conflicting threshold `5%`. The percentage of a data type = count of rows for the data type / count of rows after removing Null and NotPresent. 
    1. If there is only one candidate, then the candidate is the data type. 
    2. If there are no candidates or more than one candidate, then `String` would be the data type. 

## Examples

Assume that the count of total scanned rows is `1000`. 

1)

```golang
stats := {
    "a": {
        "NumberFloat": 1,
	"NumberInt":   999,
    },
}
```

In the above example, `NumberFloat` is considered as a noise and will not be considered in Step `3`. 

2)

```golang
stats := {
    "a": {
	"NumberInt": 999,
    },
}
```

In the above example, the column is not nullable because the count of rows that are not presented is equal to 0.1%. 

3)

```golang
stats := {
    "a": {
	"NumberInt": 900,
    },
}
```

Now, the column is nullable because we have enough rows that are not presented. 

4) 

```golang
stats := {
    "a": {
        "String":    40,
	"NumberInt": 760,
    },
}
```

The percentage of `String` (40) is 5% over the normalized count (800). So `String` is not a candidate. `NumberInt` will be the data type. 

5) 

```golang
stats := {
    "a": {
        "String":    41
	"NumberInt": 759,
    },
}
```

Now, the percentage of `String` (41) is more than 5%. So `String` is a possible candidate. Therefore, we have two conflicting candidates, so we choose `String` as the data type. 

**Note**: More examples can be found in `dynamodb/schema_test.go::TestInferDataTypes`. 

## How to run it? 

If you have AWS credentials (`~/.aws/credentials`) and config (`~/.aws/config`) in your home directory, then you need to enable config via:

```bash
export AWS_SDK_LOAD_CONFIG=true
```

Otherwise, you need to set env vars explicitly: 

```bash
export AWS_ACCESS_KEY_ID=xxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxx
export AWS_DEFAULT_REGION=xxxxxx
```

Then, you can run the following command:

```bash
go run github.com/cloudspannerecosystem/harbourbridge -driver=dynamodb

# Or, specify which tables you want to migrate: 
go run github.com/cloudspannerecosystem/harbourbridge -driver=dynamodb -tables=Table1,Table2,Table3

# Or, specify to use how many rows to determine the schema: 
go run github.com/cloudspannerecosystem/harbourbridge -driver=dynamodb -schema-sample-size=1000
```

You will see an exported schema file, like:

```sql
-- Schema generated 2020-08-16 11:35:13
--
-- Spanner schema for source table Migration
--
CREATE TABLE Migration (
    Bool BOOL,                     -- From: Bool Bool
    Bytes BYTES(MAX),              -- From: Bytes Binary
    BytesSet ARRAY<BYTES(MAX)>,    -- From: BytesSet BinarySet
    List STRING(MAX),              -- From: List List
    Map STRING(MAX),               -- From: Map Map
    Number INT64,                  -- From: Number NumberInt
    NumberSet ARRAY<INT64>,        -- From: NumberSet NumberIntSet
    String STRING(MAX),            -- From: String String
    StringSet ARRAY<STRING(MAX)>,  -- From: StringSet StringSet
    Username STRING(MAX) NOT NULL  -- From: Username String
) PRIMARY KEY (Username)
```



Patch:
diff --git a/README.md b/README.md
index 1d25800..2a4255b 100644
--- a/README.md
+++ b/README.md
@@ -57,8 +57,8 @@ anyone who can access the created database.**
 
 As it processes the PostgreSQL/MySQL data, HarbourBridge reports on progress, provides
 stats on the schema and data conversion steps, and an overall assessment of the
-quality of the conversion. It also generates a schema file and report file (and
-a bad-data file if data was dropped). See
+quality of the conversion. It also generates a schema file, report file and
+a session file (and a bad-data file if data was dropped). See
 [Files Generated by HarbourBridge](#files-generated-by-harbourbridge). Details
 of how source database's schema is mapped to Spanner can be found in the
 [Schema Conversion](#schema-conversion) section.
@@ -153,7 +153,7 @@ issues.
 
 HarbourBridge also [generates several files](#files-generated-by-harbourbridge)
 when it runs: a schema file, a report file (with detailed analysis of the
-conversion), and a bad data file (if any data was dropped).
+conversion), a session file and a bad data file (if any data was dropped).
 
 ### Sample Dump Files
 
@@ -234,7 +234,10 @@ HarbourBridge generates several files as it runs:
   schema, interspersed with comments that cross-reference to the relevant
   PostgreSQL/MySQL schema definitions.
 
-- Report file (ending in `report.txt`) contains a detailed analysis of the
+- Session file (ending in `session.json`): contains all schema and data
+  conversion state endcoded as JSON. It is basically a snapshot of the session.
+
+- Report file (ending in `report.txt`): contains a detailed analysis of the
   PostgreSQL/MySQL to Spanner migration, including table-by-table stats and an
   analysis of PostgreSQL/MySQL types that don't cleanly map onto Spanner types. Note
   that PostgreSQL/MySQL types that don't have a corresponding Spanner type are
@@ -271,6 +274,9 @@ database (plus a '.') is used.
 `-v` Specifies verbose mode. This will cause HarbourBridge to output detailed
 messages about the conversion.
 
+`-schema-only` Specifies that only schema processing will be performed.
+Any data in the source database will be ignored.
+
 ## Example Usage
 
 Details on HarbourBridge example usage for PostgreSQL and MySQL can be
diff --git a/dynamodb/schema.go b/dynamodb/schema.go
new file mode 100644
index 0000000..427a7b4
--- /dev/null
+++ b/dynamodb/schema.go
@@ -0,0 +1,327 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package dynamodb
+
+import (
+	"fmt"
+	"log"
+	"math/big"
+	"sort"
+
+	sp "cloud.google.com/go/spanner"
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/service/dynamodb"
+	"github.com/cloudspannerecosystem/harbourbridge/internal"
+	"github.com/cloudspannerecosystem/harbourbridge/schema"
+)
+
+const (
+	typeString          = "String"
+	typeBool            = "Bool"
+	typeNumber          = "Number"
+	typeNumberString    = "NumberString"
+	typeBinary          = "Binary"
+	typeList            = "List"
+	typeMap             = "Map"
+	typeStringSet       = "StringSet"
+	typeNumberSet       = "NumberSet"
+	typeNumberStringSet = "NumberStringSet"
+	typeBinarySet       = "BinarySet"
+
+	errThreshold      = float64(0.001)
+	conflictThreshold = float64(0.05)
+)
+
+type dynamoClient interface {
+	ListTables(input *dynamodb.ListTablesInput) (*dynamodb.ListTablesOutput, error)
+	DescribeTable(input *dynamodb.DescribeTableInput) (*dynamodb.DescribeTableOutput, error)
+	Scan(input *dynamodb.ScanInput) (*dynamodb.ScanOutput, error)
+}
+
+// ProcessSchema performs schema conversion for source tables in a DynamoDB
+// database. Since DynamoDB is a schemaless database, this process is imprecise.
+// We obtain schema information from two sources: from the table's metadata,
+// and from analyzing a sample of the table's rows.
+func ProcessSchema(conv *internal.Conv, client dynamoClient, tables []string, sampleSize int64) error {
+	if len(tables) == 0 {
+		var err error
+		tables, err = listTables(client)
+		if err != nil {
+			return err
+		}
+		if len(tables) == 0 {
+			return fmt.Errorf("no DynamoDB table exists under this account")
+		}
+	}
+	for _, t := range tables {
+		if err := processTable(conv, client, t, sampleSize); err != nil {
+			return err
+		}
+	}
+	schemaToDDL(conv)
+	conv.AddPrimaryKeys()
+	return nil
+}
+
+func listTables(client dynamoClient) ([]string, error) {
+	var tables []string
+	input := &dynamodb.ListTablesInput{}
+	for {
+		result, err := client.ListTables(input)
+		if err != nil {
+			return nil, err
+		}
+		for _, t := range result.TableNames {
+			tables = append(tables, *t)
+		}
+
+		if result.LastEvaluatedTableName == nil {
+			return tables, nil
+		}
+		input.ExclusiveStartTableName = result.LastEvaluatedTableName
+	}
+}
+
+func processTable(conv *internal.Conv, client dynamoClient, table string, sampleSize int64) error {
+	dySchema := schema.Table{Name: table}
+	err := analyzeMetadata(client, &dySchema)
+	if err != nil {
+		return err
+	}
+	stats, count, err := scanSampleData(client, sampleSize, dySchema.Name)
+	if err != nil {
+		return err
+	}
+	inferDataTypes(stats, count, &dySchema)
+
+	// Sort column names in increasing order, because the server may return them
+	// in a random order.
+	sort.Strings(dySchema.ColNames)
+	conv.SrcSchema[table] = dySchema
+	return nil
+}
+
+func analyzeMetadata(client dynamoClient, s *schema.Table) error {
+	input := &dynamodb.DescribeTableInput{
+		TableName: aws.String(s.Name),
+	}
+
+	result, err := client.DescribeTable(input)
+	if err != nil {
+		return fmt.Errorf("failed to make a DescribeTable API call for table %v: %v", s.Name, err)
+	}
+
+	// Primary keys
+	for _, i := range result.Table.KeySchema {
+		s.PrimaryKeys = append(s.PrimaryKeys, schema.Key{Column: *i.AttributeName})
+	}
+
+	// Secondary indexes
+	for _, i := range result.Table.GlobalSecondaryIndexes {
+		var keys []schema.Key
+		for _, j := range i.KeySchema {
+			keys = append(keys, schema.Key{Column: *j.AttributeName})
+		}
+		// s.SecIndexes = append(s.SecIndexes, index{Name: *i.IndexName, Keys: keys})
+		s.Indexes = append(s.Indexes, schema.Index{Name: *i.IndexName, Keys: keys})
+	}
+
+	return nil
+}
+
+func scanSampleData(client dynamoClient, sampleSize int64, table string) (map[string]map[string]int64, int64, error) {
+	// A map from column name to a count map of possible data types.
+	stats := make(map[string]map[string]int64)
+	var count int64
+	// Build the query input parameters
+	params := &dynamodb.ScanInput{
+		TableName: aws.String(table),
+	}
+
+	for {
+		// Make the DynamoDB Query API call
+		result, err := client.Scan(params)
+		if err != nil {
+			return nil, 0, fmt.Errorf("failed to make Query API call for table %v: %v", table, err)
+		}
+
+		// Iterate the items returned
+		for _, attrsMap := range result.Items {
+			for attrName, attr := range attrsMap {
+				if _, ok := stats[attrName]; !ok {
+					stats[attrName] = make(map[string]int64)
+				}
+				incTypeCount(attrName, attr, stats[attrName])
+			}
+
+			count++
+			if count >= sampleSize {
+				return stats, count, nil
+			}
+		}
+		if result.LastEvaluatedKey == nil {
+			break
+		}
+		// If there are more rows, then continue.
+		params.ExclusiveStartKey = result.LastEvaluatedKey
+	}
+	return stats, count, nil
+}
+
+func incTypeCount(attrName string, attr *dynamodb.AttributeValue, s map[string]int64) {
+	switch {
+	case attr.S != nil:
+		s[typeString]++
+	case attr.BOOL != nil:
+		s[typeBool]++
+	case attr.N != nil:
+		// We map the DynamoDB Number type into Spanner's NUMERIC type
+		// if it fits and STRING otherwise. Note that DyanamoDB's Number
+		// type has more precision/range than Spanner's NUMERIC.
+		// We could potentially do a more detailed analysis and see if
+		// the number fits in an INT64 or FLOAT64, but we've chosen to
+		// keep the analysis simple for the moment.
+		if numericParsable(*attr.N) {
+			s[typeNumber]++
+		} else {
+			s[typeNumberString]++
+		}
+	case len(attr.B) != 0:
+		s[typeBinary]++
+	case attr.NULL != nil:
+		// Skip, if not present, it means nullable.
+	case len(attr.L) != 0:
+		s[typeList]++
+	case len(attr.M) != 0:
+		s[typeMap]++
+	case len(attr.SS) != 0:
+		s[typeStringSet]++
+	case len(attr.NS) != 0:
+		parsable := true
+		for _, n := range attr.NS {
+			if !numericParsable(*n) {
+				parsable = false
+				break
+			}
+		}
+		if parsable {
+			s[typeNumberSet]++
+		} else {
+			s[typeNumberStringSet]++
+		}
+	case len(attr.BS) != 0:
+		s[typeBinarySet]++
+	default:
+		log.Printf("Invalid DynamoDB data type: %v - %v", attrName, attr)
+	}
+}
+
+type statItem struct {
+	Type  string
+	Count int64
+}
+
+func inferDataTypes(stats map[string]map[string]int64, rows int64, s *schema.Table) {
+	if s.ColDefs == nil {
+		s.ColDefs = make(map[string]schema.Column)
+	}
+
+	for col, countMap := range stats {
+		var statItems, candidates []statItem
+		var presentRows int64
+		for k, v := range countMap {
+			presentRows += v
+			if float64(v)/float64(rows) <= errThreshold {
+				// If the percentage is less than the error threshold, then
+				// this data type has a high chance to be mistakenly inserted
+				// and we should discard it.
+				continue
+			}
+			statItems = append(statItems, statItem{Type: k, Count: v})
+		}
+		if len(statItems) == 0 {
+			log.Printf("Skip column %v with no data records", col)
+			continue
+		}
+
+		// Check if the column is a part of a primary key.
+		isPKey := false
+		for _, pk := range s.PrimaryKeys {
+			if pk.Column == col {
+				isPKey = true
+				break
+			}
+		}
+
+		// If this column is in the primary key, then it cannot be null.
+		nullable := false
+		if !isPKey {
+			nullable = float64(rows-presentRows)/float64(rows) > errThreshold
+		}
+
+		for _, si := range statItems {
+			if float64(si.Count)/float64(presentRows) > conflictThreshold {
+				// If the normalized percentage is greater than the conflicting
+				// threshold, we should consider this data type as a candidate.
+				candidates = append(candidates, si)
+			}
+		}
+
+		s.ColNames = append(s.ColNames, col)
+		if len(candidates) == 1 {
+			s.ColDefs[col] = schema.Column{Name: col, Type: schema.Type{Name: candidates[0].Type}, NotNull: !nullable}
+		} else {
+			// If there is no any candidate or more than a single candidate,
+			// this column has a significant conflict on data types and then
+			// defaults to a String type.
+			s.ColDefs[col] = schema.Column{Name: col, Type: schema.Type{Name: typeString}, NotNull: !nullable}
+		}
+	}
+}
+
+// numericParsable determines whether its argument is a valid Spanner numeric
+// values. This is based on the definition of the NUMERIC type in Cloud Spanner:
+// a NUMERIC type with 38 digits of precision and 9 digits of scale. It can
+// support 29 digits before the decimal point and 9 digits after that.
+func numericParsable(n string) bool {
+	y, ok := (&big.Rat{}).SetString(n)
+	if !ok {
+		return false
+	}
+	// Get the length of numerator in text (base-10).
+	numLen := len(y.Num().Text(10))
+	// Remove the sign `-` if it exists.
+	if y.Num().Sign() == -1 {
+		numLen--
+	}
+	if numLen > sp.NumericPrecisionDigits {
+		return false
+	}
+
+	// Get the length of denominator in text (base-10). Remove a digit because
+	// the length of denominator would have one mor digit than the expected
+	// scale. E.g., 0.999 will become 999/1000 and the length of denominator is
+	// 4 instead of 3.
+	denomLen := len(y.Denom().Text(10)) - 1
+	// Remove the sign `-` if it exists.
+	if y.Denom().Sign() == -1 {
+		denomLen--
+	}
+	if denomLen > sp.NumericScaleDigits {
+		return false
+	}
+
+	return true
+}
diff --git a/dynamodb/toddl.go b/dynamodb/toddl.go
new file mode 100644
index 0000000..518f6df
--- /dev/null
+++ b/dynamodb/toddl.go
@@ -0,0 +1,124 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package dynamodb
+
+import (
+	"fmt"
+	"strconv"
+	"unicode"
+
+	"github.com/cloudspannerecosystem/harbourbridge/internal"
+	"github.com/cloudspannerecosystem/harbourbridge/schema"
+	"github.com/cloudspannerecosystem/harbourbridge/spanner/ddl"
+)
+
+// schemaToDDL performs schema conversion from the source DB schema to
+// Spanner. It uses the source schema in conv.SrcSchema, and writes
+// the Spanner schema to conv.SpSchema.
+func schemaToDDL(conv *internal.Conv) error {
+	for _, srcTable := range conv.SrcSchema {
+		spTableName, err := internal.GetSpannerTable(conv, srcTable.Name)
+		if err != nil {
+			conv.Unexpected(fmt.Sprintf("Couldn't map source table %s to Spanner: %s", srcTable.Name, err))
+			continue
+		}
+		var spColNames []string
+		spColDef := make(map[string]ddl.ColumnDef)
+		conv.Issues[srcTable.Name] = make(map[string][]internal.SchemaIssue)
+		// Iterate over columns using ColNames order.
+		for _, srcColName := range srcTable.ColNames {
+			srcCol := srcTable.ColDefs[srcColName]
+			colName, err := internal.GetSpannerCol(conv, srcTable.Name, srcCol.Name, false)
+			if err != nil {
+				conv.Unexpected(fmt.Sprintf("Couldn't map source column %s of table %s to Spanner: %s", srcTable.Name, srcCol.Name, err))
+				continue
+			}
+			spColNames = append(spColNames, colName)
+			ty, issues := toSpannerType(conv, srcCol.Type.Name, srcCol.Type.Mods)
+
+			if len(issues) > 0 {
+				conv.Issues[srcTable.Name][srcCol.Name] = issues
+			}
+			spColDef[colName] = ddl.ColumnDef{
+				Name:    colName,
+				T:       ty,
+				NotNull: srcCol.NotNull,
+				Comment: "From: " + quoteIfNeeded(srcCol.Name) + " " + srcCol.Type.Print(),
+			}
+		}
+		comment := "Spanner schema for source table " + quoteIfNeeded(srcTable.Name)
+		conv.SpSchema[spTableName] = ddl.CreateTable{
+			Name:     spTableName,
+			ColNames: spColNames,
+			ColDefs:  spColDef,
+			Pks:      cvtPrimaryKeys(conv, srcTable.Name, srcTable.PrimaryKeys),
+			Comment:  comment}
+	}
+	return nil
+}
+
+// toSpannerType maps a scalar source schema type (defined by id and
+// mods) into a Spanner type. This is the core source-to-Spanner type
+// mapping.  toSpannerType returns the Spanner type and a list of type
+// conversion issues encountered.
+func toSpannerType(conv *internal.Conv, id string, mods []int64) (ddl.Type, []internal.SchemaIssue) {
+	maxExpectedMods := func(n int) {
+		if len(mods) > n {
+			conv.Unexpected(fmt.Sprintf("Found %d mods while processing type id=%s", len(mods), id))
+		}
+	}
+	maxExpectedMods(0)
+	switch id {
+	case typeNumber:
+		return ddl.Type{Name: ddl.Numeric}, nil
+	case typeNumberString, typeString, typeList, typeMap:
+		return ddl.Type{Name: ddl.String, Len: ddl.MaxLength}, nil
+	case typeBool:
+		return ddl.Type{Name: ddl.Bool}, nil
+	case typeBinary:
+		return ddl.Type{Name: ddl.Bytes, Len: ddl.MaxLength}, nil
+	case typeStringSet, typeNumberStringSet:
+		return ddl.Type{Name: ddl.String, Len: ddl.MaxLength, IsArray: true}, nil
+	case typeNumberSet:
+		return ddl.Type{Name: ddl.Numeric, IsArray: true}, nil
+	case typeBinarySet:
+		return ddl.Type{Name: ddl.Bytes, Len: ddl.MaxLength, IsArray: true}, nil
+	default:
+		return ddl.Type{Name: ddl.String, Len: ddl.MaxLength}, []internal.SchemaIssue{internal.NoGoodType}
+	}
+}
+
+func quoteIfNeeded(s string) string {
+	for _, r := range s {
+		if unicode.IsLetter(r) || unicode.IsDigit(r) || unicode.IsPunct(r) {
+			continue
+		}
+		return strconv.Quote(s)
+	}
+	return s
+}
+
+func cvtPrimaryKeys(conv *internal.Conv, srcTable string, srcKeys []schema.Key) []ddl.IndexKey {
+	var spKeys []ddl.IndexKey
+	for _, k := range srcKeys {
+		spCol, err := internal.GetSpannerCol(conv, srcTable, k.Column, true)
+		if err != nil {
+			conv.Unexpected(fmt.Sprintf("Can't map key for table %s", srcTable))
+			continue
+		}
+		spKeys = append(spKeys, ddl.IndexKey{Col: spCol, Desc: k.Desc})
+	}
+	return spKeys
+}
diff --git a/go.mod b/go.mod
index 8a872f2..35ee6eb 100644
--- a/go.mod
+++ b/go.mod
@@ -3,9 +3,10 @@ module github.com/cloudspannerecosystem/harbourbridge
 go 1.13
 
 require (
-	cloud.google.com/go v0.60.0
-	cloud.google.com/go/spanner v1.7.0
+	cloud.google.com/go v0.65.0
+	cloud.google.com/go/spanner v1.10.0
 	github.com/DATA-DOG/go-sqlmock v1.4.1
+	github.com/aws/aws-sdk-go v1.34.5
 	github.com/davecgh/go-spew v1.1.1
 	github.com/go-sql-driver/mysql v1.5.0
 	github.com/gorilla/handlers v1.5.0
@@ -20,9 +21,10 @@ require (
 	github.com/pingcap/tidb v1.1.0-beta.0.20200423105559-af376db3dc46
 	github.com/pingcap/tipb v0.0.0-20200417094153-7316d94df1ee // indirect
 	github.com/sirupsen/logrus v1.5.0 // indirect
-	github.com/stretchr/testify v1.4.0
-	golang.org/x/crypto v0.0.0-20191206172530-e9b2fee46413
-	google.golang.org/api v0.28.0
-	google.golang.org/genproto v0.0.0-20200626011028-ee7919e894b5
-	google.golang.org/grpc v1.29.1
+	github.com/stretchr/testify v1.5.1
+	golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
+	golang.org/x/sys v0.0.0-20200916084744-dbad9cb7cb7a // indirect
+	google.golang.org/api v0.31.0
+	google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d
+	google.golang.org/grpc v1.31.1
 )
diff --git a/go.sum b/go.sum
index 6c3f3d0..78348df 100644
--- a/go.sum
+++ b/go.sum
@@ -18,11 +18,15 @@ cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKV
 cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs=
 cloud.google.com/go v0.60.0 h1:R+tDlceO7Ss+zyvtsdhTxacDyZ1k99xwskQ4FT7ruoM=
 cloud.google.com/go v0.60.0/go.mod h1:yw2G51M9IfRboUH61Us8GqCeF1PzPblB823Mn2q2eAU=
+cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc=
+cloud.google.com/go v0.65.0 h1:Dg9iHVQfrhq82rUNu9ZxUDrJLaxFUe/HlCVaLyRruq8=
+cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY=
 cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
 cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
 cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc=
 cloud.google.com/go/bigquery v1.5.0/go.mod h1:snEHRnqQbz117VIFhE8bmtwIDY80NLUZUMb4Nv6dBIg=
 cloud.google.com/go/bigquery v1.7.0/go.mod h1://okPTzCYNXSlb24MZs83e2Do+h+VXtc4gLoIoXIAPc=
+cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM77hZzJN/fQ=
 cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
 cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk=
 cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
@@ -37,10 +41,13 @@ cloud.google.com/go/spanner v1.4.0 h1:gvmMlYl98fd0N3k16oFigS84giz0Z9OEkxak6PhqSY
 cloud.google.com/go/spanner v1.4.0/go.mod h1:7nV8byyEzhYJ/rVoqWDja7oF/IYQydFkZKoo8pynWTQ=
 cloud.google.com/go/spanner v1.7.0 h1:mvgDB+f4CfKXebTIi36aqY02eVG4nU/Z3KV6stQaYCc=
 cloud.google.com/go/spanner v1.7.0/go.mod h1:sd3K2gZ9Fd0vMPLXzeCrF6fq4i63Q7aTLW/lBIfBkIk=
+cloud.google.com/go/spanner v1.10.0 h1:B2I4WJY5DyY2LW1iCeY+23hCTyWlsQWsupKlTW7l6ws=
+cloud.google.com/go/spanner v1.10.0/go.mod h1:eVM2ifUL7rm+wM4gAXMuwO8CpNjbAJXK7CKZjUtt8Jw=
 cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
 cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
 cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk=
 cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs=
+cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
 github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
@@ -60,6 +67,8 @@ github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRF
 github.com/antihax/optional v0.0.0-20180407024304-ca021399b1a6/go.mod h1:V8iCPQYkqmusNa815XgQio277wI47sdRh1dUOLdyC6Q=
 github.com/appleboy/gin-jwt/v2 v2.6.3/go.mod h1:MfPYA4ogzvOcVkRwAxT7quHOtQmVKDpTwxyUrC2DNw0=
 github.com/appleboy/gofight/v2 v2.1.2/go.mod h1:frW+U1QZEdDgixycTj4CygQ48yLTUhplt43+Wczp3rw=
+github.com/aws/aws-sdk-go v1.34.5 h1:FwubVVX9u+kW9qDCjVzyWOdsL+W5wPq683wMk2R2GXk=
+github.com/aws/aws-sdk-go v1.34.5/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU8YkpAk0=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
 github.com/beorn7/perks v1.0.0 h1:HWo1m869IqiPhD389kmkxeTalrjNbbJTC8LXupb+sl0=
 github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
@@ -189,6 +198,7 @@ github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFU
 github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
 github.com/golang/mock v1.4.1/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
 github.com/golang/mock v1.4.3/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
+github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
 github.com/golang/protobuf v0.0.0-20180814211427-aa810b61a9c7/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
@@ -217,10 +227,15 @@ github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMyw
 github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
 github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4=
 github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.4.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.0 h1:/QaMHBdZ26BB3SSst0Iwl10Epc+xhTquomWX0oZEB6w=
 github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM=
+github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
 github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
+github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
 github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
 github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
 github.com/google/pprof v0.0.0-20190930153522-6ce02741cba3/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
@@ -230,6 +245,7 @@ github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3 h1:SRgJV+IoxM5MKyFdlS
 github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
 github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
 github.com/google/pprof v0.0.0-20200507031123-427632fa3b1c/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
+github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
 github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
 github.com/google/shlex v0.0.0-20181106134648-c34317bd91bf/go.mod h1:RpwtwJQFrIEPstU94h88MWPXP2ektJZ8cZ0YntAmXiE=
 github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
@@ -267,6 +283,8 @@ github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANyt
 github.com/jinzhu/gorm v1.9.12/go.mod h1:vhTjlKSJUTWNtcbQtrMBFCxy7eXTzeCAzfL5fBZT/Qs=
 github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc=
 github.com/jinzhu/now v1.0.1/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
+github.com/jmespath/go-jmespath v0.3.0 h1:OS12ieG61fsCg5+qLJ+SsW9NicxNkg3b25OyT2yCeUc=
+github.com/jmespath/go-jmespath v0.3.0/go.mod h1:9QtRXoHjLGCJ5IBSaohpXITPlowMeeYCZ7fLUTSywik=
 github.com/joho/godotenv v1.3.0/go.mod h1:7hK45KPybAkOC6peb+G5yklZfMxEjkZhHbwpqxOKXbg=
 github.com/jonboulle/clockwork v0.1.0 h1:VKV+ZcuP6l3yW9doeqz6ziZGgcynBVQO+obU0+0hcPo=
 github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
@@ -449,6 +467,8 @@ github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXf
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
 github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4=
+github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
 github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14/go.mod h1:gxQT6pBGRuIGunNf/+tSOB5OHvguWi8Tbt82WOkf35E=
 github.com/swaggo/gin-swagger v1.2.0/go.mod h1:qlH2+W7zXGZkczuL+r2nEBR2JTT+/lX05Nn6vPhc7OI=
 github.com/swaggo/http-swagger v0.0.0-20200103000832-0e9263c4b516/go.mod h1:O1lAbCgAAX/KZ80LM/OXwtWFI/5TvZlwxSg8Cq08PV0=
@@ -489,6 +509,8 @@ github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q
 github.com/yookoala/realpath v1.0.0/go.mod h1:gJJMA9wuX7AcqLy1+ffPatSCySA1FQ2S8Ya9AIoYBpE=
 github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
+github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
+github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 go.etcd.io/bbolt v1.3.3 h1:MUGmc65QhB3pIlaQ5bB4LwqSj6GIonVJXpZiaKNyaKk=
 go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
 go.etcd.io/etcd v0.0.0-20191023171146-3cf2f69b5738/go.mod h1:dnLIgRNXwCJa5e+c6mIZCrds/GIG4ncV9HhK5PX7jPg=
@@ -500,6 +522,8 @@ go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
 go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
 go.opencensus.io v0.22.3 h1:8sGtKOrtQqkN1bp2AtX+misvLIlOmsEsNd+9NIcPEm8=
 go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
+go.opencensus.io v0.22.4 h1:LYy1Hy3MJdrCdMwwzxA/dRok4ejH+RwNGbuoD9fCjto=
+go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
 go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
 go.uber.org/atomic v1.5.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
 go.uber.org/atomic v1.6.0 h1:Ezj3JGmsOnG1MoRWQkPBsKLe9DwWD9QeXzTRzzldNVk=
@@ -532,6 +556,8 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U
 golang.org/x/crypto v0.0.0-20191205180655-e7c4368fe9dd/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20191206172530-e9b2fee46413 h1:ULYEB3JvPRE/IfO+9uO7vKV/xzVTO7XPAwm8xbf4w2g=
 golang.org/x/crypto v0.0.0-20191206172530-e9b2fee46413/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI=
+golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -584,6 +610,7 @@ golang.org/x/net v0.0.0-20190611141213-3f473d35a33a/go.mod h1:z5CRVTTTmAJ677TzLL
 golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859 h1:R/3boaszxrf1GEUWTVDzSKVwLmSJpwZ1yqXm8j0v2QI=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20190628185345-da137c7871d7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
@@ -599,8 +626,13 @@ golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e h1:3G+cUijn7XD+S4eJFddp53Pv7
 golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
 golang.org/x/net v0.0.0-20200501053045-e0ff5e5a1de5/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
 golang.org/x/net v0.0.0-20200506145744-7e3656a0809f/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
+golang.org/x/net v0.0.0-20200513185701-a91f0712d120/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
 golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2 h1:eDrdRpKgkcCqKZQwyZRyeFZgfqt37SL7Kv3tok06cKE=
 golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
+golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
+golang.org/x/net v0.0.0-20200822124328-c89045814202 h1:VvcQYSHwXgi7W+TpUR6A9g6Up98WAHf3f/ulnJ62IyA=
+golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45 h1:SVwTIAaPC2U/AvvLNZ2a7OVsmBpC8L5BlwK1whH3hm0=
@@ -616,6 +648,7 @@ golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a h1:WXEvlFVvvGxCJLG6REjsT03iWnKLEWinaScsxF2Vm2o=
 golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -654,13 +687,21 @@ golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20200331124033-c3d80250170d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200501052902-10377860bb8e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200511232937-7e40ca221e25/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200523222454-059865788121 h1:rITEj+UZHYC927n8GT97eC3zrpzXdb/voyeOuVKS46o=
 golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200828194041-157a740278f4 h1:kCCpuwSAoYJPkNc6x0xT9yTtV4oKtARo4RGBQWOfg9E=
+golang.org/x/sys v0.0.0-20200828194041-157a740278f4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200916084744-dbad9cb7cb7a h1:chkwkn8HYWVtTE5DCQNKYlkyptadXYY0+PuyaVdyMo4=
+golang.org/x/sys v0.0.0-20200916084744-dbad9cb7cb7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
+golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
@@ -721,13 +762,23 @@ golang.org/x/tools v0.0.0-20200325203130-f53864d0dba1/go.mod h1:Sl4aGygMT6LrqrWc
 golang.org/x/tools v0.0.0-20200331025713-a30bf2db82d4/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8=
 golang.org/x/tools v0.0.0-20200501065659-ab2804fb9c9d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
 golang.org/x/tools v0.0.0-20200512131952-2bc93b1c0c88/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
+golang.org/x/tools v0.0.0-20200515010526-7d3b6ebf133d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
+golang.org/x/tools v0.0.0-20200618134242-20370b0cb4b2/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
 golang.org/x/tools v0.0.0-20200626171337-aa94e735be7f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
 golang.org/x/tools v0.0.0-20200630154851-b2d8b0336632 h1:fxWeLV4ol1icb47+btwfzuf8gOkJE/PPo9bLjSvZzA8=
 golang.org/x/tools v0.0.0-20200630154851-b2d8b0336632/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
+golang.org/x/tools v0.0.0-20200729194436-6467de6f59a7/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
+golang.org/x/tools v0.0.0-20200804011535-6c149bb5ef0d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
+golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
+golang.org/x/tools v0.0.0-20200828161849-5deb26317202/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
+golang.org/x/tools v0.0.0-20200903185744-af4cc2cd812e h1:RvNtqusJ+6DJ07/by/M84a6/Dd17XU6n8QvhvknjJno=
+golang.org/x/tools v0.0.0-20200903185744-af4cc2cd812e/go.mod h1:Cj7w3i3Rnn0Xh82ur9kSqwfTHTeVxaDqrfMjpcNT6bE=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
+golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
 google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M=
 google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
@@ -745,6 +796,10 @@ google.golang.org/api v0.22.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/
 google.golang.org/api v0.24.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE=
 google.golang.org/api v0.28.0 h1:jMF5hhVfMkTZwHW1SDpKq5CkgWLXOb31Foaca9Zr3oM=
 google.golang.org/api v0.28.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE=
+google.golang.org/api v0.29.0/go.mod h1:Lcubydp8VUV7KeIHD9z2Bys/sm/vGKnG1UHuDBSrHWM=
+google.golang.org/api v0.30.0/go.mod h1:QGmEvQ87FHZNiUVJkT14jQNYJ4ZJjdRF23ZXz5138Fc=
+google.golang.org/api v0.31.0 h1:1w5Sz/puhxFo9lTtip2n47k7toB/U2nCqOKNHd3Yrbo=
+google.golang.org/api v0.31.0/go.mod h1:CL+9IBCa2WWU6gRuBWaKqGWLFFwbEUXkfeMkHLQWYWo=
 google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
 google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
@@ -781,9 +836,17 @@ google.golang.org/genproto v0.0.0-20200318110522-7735f76e9fa5/go.mod h1:55QSHmfG
 google.golang.org/genproto v0.0.0-20200331122359-1ee6d9798940/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
 google.golang.org/genproto v0.0.0-20200430143042-b979b6f78d84/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
 google.golang.org/genproto v0.0.0-20200511104702-f5ebc3bea380/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
+google.golang.org/genproto v0.0.0-20200515170657-fc4c6c6a6587/go.mod h1:YsZOwe1myG/8QRHRsmBRE1LrgQY60beZKjly0O1fX9U=
 google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
+google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7FcilCzHH/e9qn6dsT145K34l5v+OpcnNgKAAA=
 google.golang.org/genproto v0.0.0-20200626011028-ee7919e894b5 h1:a/Sqq5B3dGnmxhuJZIHFsIxhEkqElErr5TaU6IqBAj0=
 google.golang.org/genproto v0.0.0-20200626011028-ee7919e894b5/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
+google.golang.org/genproto v0.0.0-20200729003335-053ba62fc06f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
+google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
+google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
+google.golang.org/genproto v0.0.0-20200831141814-d751682dd103/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
+google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d h1:92D1fum1bJLKSdr11OJ+54YeCMCGYIygTA7R/YZxH5M=
+google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
 google.golang.org/grpc v0.0.0-20180607172857-7a6a684ca69e/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
 google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
@@ -800,6 +863,10 @@ google.golang.org/grpc v1.28.0 h1:bO/TA4OxCOummhSf10siHuG7vJOiwh7SpRpFZDkOgl4=
 google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60=
 google.golang.org/grpc v1.29.1 h1:EC2SB8S04d2r73uptxphDSUG+kTKVgjRPF+N3xpxRB4=
 google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk=
+google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
+google.golang.org/grpc v1.31.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
+google.golang.org/grpc v1.31.1 h1:SfXqXS5hkufcdZ/mHtYCh53P2b+92WQq/DZcKLgsFRs=
+google.golang.org/grpc v1.31.1/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
 google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
 google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
 google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
@@ -846,6 +913,7 @@ honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWh
 honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
 honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
+honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
 k8s.io/klog v1.0.0/go.mod h1:4Bi6QPql/J/LkTDqv7R/cd3hPo4k2DG6Ptcz060Ez5I=
 rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
 rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
diff --git a/internal/convert.go b/internal/convert.go
index b62df6b..16c6188 100644
--- a/internal/convert.go
+++ b/internal/convert.go
@@ -113,9 +113,9 @@ type stats struct {
 }
 
 type statementStat struct {
-	schema int64
-	data   int64
-	skip   int64
+	Schema int64
+	Data   int64
+	Skip   int64
 	Error  int64
 }
 
@@ -219,7 +219,7 @@ func (conv *Conv) BadRows() int64 {
 func (conv *Conv) Statements() int64 {
 	n := int64(0)
 	for _, x := range conv.Stats.Statement {
-		n += x.schema + x.data + x.skip + x.Error
+		n += x.Schema + x.Data + x.Skip + x.Error
 	}
 	return n
 }
@@ -356,7 +356,7 @@ func (conv *Conv) getStatementStat(s string) *statementStat {
 func (conv *Conv) SkipStatement(stmtType string) {
 	if conv.SchemaMode() { // Record statement stats on first pass only.
 		VerbosePrintf("Skipping statement: %s\n", stmtType)
-		conv.getStatementStat(stmtType).skip++
+		conv.getStatementStat(stmtType).Skip++
 	}
 }
 
@@ -371,14 +371,14 @@ func (conv *Conv) ErrorInStatement(stmtType string) {
 // SchemaStatement increments the schema statement stats for 'stmtType'.
 func (conv *Conv) SchemaStatement(stmtType string) {
 	if conv.SchemaMode() { // Record statement stats on first pass only.
-		conv.getStatementStat(stmtType).schema++
+		conv.getStatementStat(stmtType).Schema++
 	}
 }
 
 // DataStatement increments the data statement stats for 'stmtType'.
 func (conv *Conv) DataStatement(stmtType string) {
 	if conv.SchemaMode() { // Record statement stats on first pass only.
-		conv.getStatementStat(stmtType).data++
+		conv.getStatementStat(stmtType).Data++
 	}
 }
 
diff --git a/internal/report.go b/internal/report.go
index 706ec4f..accc56a 100644
--- a/internal/report.go
+++ b/internal/report.go
@@ -62,7 +62,7 @@ func GenerateReport(driverName string, conv *Conv, w *bufio.Writer, badWrites ma
 			h = h + fmt.Sprintf(" (mapped to Spanner table %s)", t.spTable)
 		}
 		writeHeading(w, h)
-		w.WriteString(rateConversion(t.rows, t.badRows, t.cols, t.warnings, t.syntheticPKey != "", false))
+		w.WriteString(rateConversion(t.rows, t.badRows, t.cols, t.warnings, t.syntheticPKey != "", false, conv.SchemaMode()))
 		w.WriteString("\n")
 		for _, x := range t.body {
 			fmt.Fprintf(w, "%s\n", x.heading)
@@ -126,7 +126,9 @@ func buildTableReport(conv *Conv, srcTable string, badWrites map[string]int64) t
 	} else {
 		tr.body = buildTableReportBody(conv, srcTable, issues, spSchema, srcSchema, nil)
 	}
-	fillRowStats(conv, srcTable, badWrites, &tr)
+	if !conv.SchemaMode() {
+		fillRowStats(conv, srcTable, badWrites, &tr)
+	}
 	return tr
 }
 
@@ -187,7 +189,7 @@ func buildTableReportBody(conv *Conv, srcTable string, issues map[string][]Schem
 				case DefaultValue:
 					l = append(l, fmt.Sprintf("%s e.g. column '%s'", issueDB[i].brief, srcCol))
 				case ForeignKey:
-					l = append(l, fmt.Sprintf("Column '%s' uses foreign keys which Spanner does not support", srcCol))
+					l = append(l, fmt.Sprintf("Column '%s' uses foreign keys which HarbourBridge does not support yet", srcCol))
 				case AutoIncrement:
 					l = append(l, fmt.Sprintf("Column '%s' is an autoincrement column. %s", srcCol, issueDB[i].brief))
 				case Timestamp:
@@ -355,9 +357,12 @@ func ok(total, badCount int64) bool {
 	return badCount < total/3
 }
 
-func rateConversion(rows, badRows, cols, warnings int64, missingPKey, summary bool) string {
-	return fmt.Sprintf("Schema conversion: %s.\n", rateSchema(cols, warnings, missingPKey, summary)) +
-		fmt.Sprintf("Data conversion: %s.\n", rateData(rows, badRows))
+func rateConversion(rows, badRows, cols, warnings int64, missingPKey, summary bool, schemaOnly bool) string {
+	rate := fmt.Sprintf("Schema conversion: %s.\n", rateSchema(cols, warnings, missingPKey, summary))
+	if !schemaOnly {
+		rate = rate + fmt.Sprintf("Data conversion: %s.\n", rateData(rows, badRows))
+	}
+	return rate
 }
 
 func generateSummary(conv *Conv, r []tableReport, badWrites map[string]int64) string {
@@ -385,7 +390,7 @@ func generateSummary(conv *Conv, r []tableReport, badWrites map[string]int64) st
 	for _, n := range badWrites {
 		badRows += n
 	}
-	return rateConversion(rows, badRows, cols, warnings, missingPKey, true)
+	return rateConversion(rows, badRows, cols, warnings, missingPKey, true, conv.SchemaMode())
 }
 
 func ignoredStatements(conv *Conv) (l []string) {
@@ -416,7 +421,7 @@ func writeStmtStats(driverName string, conv *Conv, w *bufio.Writer) {
 	}
 	var l []stat
 	for s, x := range conv.Stats.Statement {
-		l = append(l, stat{s, x.schema + x.data + x.skip + x.Error})
+		l = append(l, stat{s, x.Schema + x.Data + x.Skip + x.Error})
 	}
 	// Sort by alphabetical order of statements.
 	sort.Slice(l, func(i, j int) bool {
@@ -433,7 +438,7 @@ func writeStmtStats(driverName string, conv *Conv, w *bufio.Writer) {
 	w.WriteString("  --------------------------------------\n")
 	for _, x := range l {
 		s := conv.Stats.Statement[x.statement]
-		fmt.Fprintf(w, "  %6d %6d %6d %6d  %s\n", s.schema, s.data, s.skip, s.Error, x.statement)
+		fmt.Fprintf(w, "  %6d %6d %6d %6d  %s\n", s.Schema, s.Data, s.Skip, s.Error, x.statement)
 	}
 	if driverName == "pg_dump" {
 		w.WriteString("See github.com/lfittl/pg_query_go/nodes for definitions of statement types\n")
diff --git a/main.go b/main.go
index af68daf..cc0f229 100644
--- a/main.go
+++ b/main.go
@@ -22,6 +22,7 @@ import (
 	"context"
 	"crypto/rand"
 	"database/sql"
+	"encoding/json"
 	"flag"
 	"fmt"
 	"io"
@@ -37,6 +38,8 @@ import (
 	sp "cloud.google.com/go/spanner"
 	database "cloud.google.com/go/spanner/admin/database/apiv1"
 	instance "cloud.google.com/go/spanner/admin/instance/apiv1"
+	"github.com/aws/aws-sdk-go/aws/session"
+	dydb "github.com/aws/aws-sdk-go/service/dynamodb"
 	_ "github.com/go-sql-driver/mysql"
 	_ "github.com/lib/pq"
 	"golang.org/x/crypto/ssh/terminal"
@@ -44,6 +47,7 @@ import (
 	adminpb "google.golang.org/genproto/googleapis/spanner/admin/database/v1"
 	instancepb "google.golang.org/genproto/googleapis/spanner/admin/instance/v1"
 
+	"github.com/cloudspannerecosystem/harbourbridge/dynamodb"
 	"github.com/cloudspannerecosystem/harbourbridge/internal"
 	"github.com/cloudspannerecosystem/harbourbridge/mysql"
 	"github.com/cloudspannerecosystem/harbourbridge/postgres"
@@ -60,17 +64,23 @@ const (
 	MYSQLDUMP string = "mysqldump"
 	// MYSQL is the driver name for MySQL.
 	MYSQL string = "mysql"
+	// DYNAMODB is the driver name for AWS DynamoDB.
+	// This is an experimental driver; implementation in progress.
+	DYNAMODB string = "dynamodb"
 )
 
 var (
 	badDataFile      = "dropped.txt"
 	schemaFile       = "schema.txt"
+	sessionFile      = "session.json"
 	reportFile       = "report.txt"
 	dbNameOverride   string
 	instanceOverride string
 	filePrefix       = ""
 	driverName       = PGDUMP
+	schemaSampleSize = int64(0)
 	verbose          bool
+	schemaOnly       bool
 )
 
 func init() {
@@ -78,7 +88,10 @@ func init() {
 	flag.StringVar(&instanceOverride, "instance", "", "instance: Spanner instance to use")
 	flag.StringVar(&filePrefix, "prefix", "", "prefix: file prefix for generated files")
 	flag.StringVar(&driverName, "driver", "pg_dump", "driver name: flag for accessing source DB or dump files (accepted values are \"pg_dump\", \"postgres\", \"mysqldump\", and \"mysql\")")
+	flag.Int64Var(&schemaSampleSize, "schema-sample-size", int64(100000), "schema-sample-size: the number of rows to use for inferring schema (only for DynamoDB)")
 	flag.BoolVar(&verbose, "v", false, "verbose: print additional output")
+	flag.BoolVar(&schemaOnly, "schema-only", false, "schema-only: mode for schema only conversion to Spanner")
+
 }
 
 func usage() {
@@ -104,23 +117,26 @@ func main() {
 
 	ioHelper := &ioStreams{in: os.Stdin, out: os.Stdout}
 	fmt.Println("Using driver (source DB):", driverName)
-	project, err := getProject()
-	if err != nil {
-		fmt.Printf("\nCan't get project: %v\n", err)
-		panic(fmt.Errorf("can't get project"))
-	}
-	fmt.Println("Using Google Cloud project:", project)
-
-	instance := instanceOverride
-	if instance == "" {
-		instance, err = getInstance(project, ioHelper.out)
+	var project, instance string
+	if !schemaOnly {
+		project, err = getProject()
 		if err != nil {
-			fmt.Printf("\nCan't get instance: %v\n", err)
-			panic(fmt.Errorf("can't get instance"))
+			fmt.Printf("\nCan't get project: %v\n", err)
+			panic(fmt.Errorf("can't get project"))
+		}
+		fmt.Println("Using Google Cloud project:", project)
+
+		instance = instanceOverride
+		if instance == "" {
+			instance, err = getInstance(project, ioHelper.out)
+			if err != nil {
+				fmt.Printf("\nCan't get instance: %v\n", err)
+				panic(fmt.Errorf("can't get instance"))
+			}
 		}
+		fmt.Println("Using Cloud Spanner instance:", instance)
+		printPermissionsWarning(driverName, ioHelper.out)
 	}
-	fmt.Println("Using Cloud Spanner instance:", instance)
-	printPermissionsWarning(driverName, ioHelper.out)
 
 	now := time.Now()
 	dbName := dbNameOverride
@@ -162,6 +178,16 @@ func toSpanner(driver, projectID, instanceID, dbName string, ioHelper *ioStreams
 	}
 
 	writeSchemaFile(conv, now, outputFilePrefix+schemaFile, ioHelper.out)
+	writeSessionFile(conv, now, outputFilePrefix+sessionFile, ioHelper.out)
+	if schemaOnly {
+		report(driver, nil, ioHelper.bytesRead, "", conv, outputFilePrefix+reportFile, ioHelper.out)
+		return nil
+	}
+
+	// TODO(hengfeng): delete the following code after data conversion is done.
+	if driver == DYNAMODB {
+		return nil
+	}
 
 	db, err := createDatabase(projectID, instanceID, dbName, conv, ioHelper.out)
 	if err != nil {
@@ -192,6 +218,8 @@ func schemaConv(driver string, ioHelper *ioStreams) (*internal.Conv, error) {
 		return schemaFromSQL(driver)
 	case PGDUMP, MYSQLDUMP:
 		return schemaFromDump(driver, ioHelper)
+	case DYNAMODB:
+		return schemaFromDynamoDB(schemaSampleSize)
 	default:
 		return nil, fmt.Errorf("schema conversion for driver %s not supported", driver)
 	}
@@ -318,6 +346,17 @@ func dataFromSQL(driver string, config spanner.BatchWriterConfig, client *sp.Cli
 	return writer, nil
 }
 
+func schemaFromDynamoDB(sampleSize int64) (*internal.Conv, error) {
+	conv := internal.MakeConv()
+	mySession := session.Must(session.NewSession())
+	client := dydb.New(mySession)
+	err := dynamodb.ProcessSchema(conv, client, []string{}, sampleSize)
+	if err != nil {
+		return nil, err
+	}
+	return conv, nil
+}
+
 type ioStreams struct {
 	in, seekableIn, out *os.File
 	bytesRead           int64
@@ -567,6 +606,26 @@ func writeSchemaFile(conv *internal.Conv, now time.Time, name string, out *os.Fi
 	fmt.Fprintf(out, "Wrote schema to file '%s'.\n", name)
 }
 
+func writeSessionFile(conv *internal.Conv, now time.Time, name string, out *os.File) {
+	f, err := os.Create(name)
+	if err != nil {
+		fmt.Fprintf(out, "Can't create session file %s: %v\n", name, err)
+		return
+	}
+	// Session file will basically contain 'conv' struct in JSON format.
+	// It contains all the information for schema and data conversion state.
+	convJSON, err := json.MarshalIndent(conv, "", " ")
+	if err != nil {
+		fmt.Fprintf(out, "Can't encode session state to JSON: %v\n", err)
+		return
+	}
+	if _, err := f.Write(convJSON); err != nil {
+		fmt.Fprintf(out, "Can't write out session file: %v\n", err)
+		return
+	}
+	fmt.Fprintf(out, "Wrote session to file '%s'.\n", name)
+}
+
 // writeBadData prints summary stats about bad rows and writes detailed info
 // to file 'name'.
 func writeBadData(bw *spanner.BatchWriter, conv *internal.Conv, banner, name string, out *os.File) {
diff --git a/postgres/pgdump.go b/postgres/pgdump.go
index 7e203a9..aa2e5df 100644
--- a/postgres/pgdump.go
+++ b/postgres/pgdump.go
@@ -62,7 +62,13 @@ func ProcessPgDump(conv *internal.Conv, r *internal.Reader) error {
 			case copyFrom:
 				processCopyBlock(conv, ci.table, ci.cols, r)
 			case insert:
-				ProcessDataRow(conv, ci.table, ci.cols, ci.vals)
+				// Handle INSERT statements where columns are not
+				// specified i.e. an insert for all table columns.
+				if len(ci.cols) == 0 {
+					ProcessDataRow(conv, ci.table, conv.SrcSchema[ci.table].ColNames, ci.vals)
+				} else {
+					ProcessDataRow(conv, ci.table, ci.cols, ci.vals)
+				}
 			}
 		}
 		if r.EOF {
diff --git a/spanner/ddl/ast.go b/spanner/ddl/ast.go
index c27382a..6e897da 100644
--- a/spanner/ddl/ast.go
+++ b/spanner/ddl/ast.go
@@ -42,13 +42,15 @@ const (
 	String string = "STRING"
 	// Timestamp represent TIMESTAMP type.
 	Timestamp string = "TIMESTAMP"
+	// Numeric represent NUMERIC type.
+	Numeric string = "NUMERIC"
 	// MaxLength is a sentinel for Type's Len field, representing the MAX value.
 	MaxLength = math.MaxInt64
 )
 
 // Type represents the type of a column.
 //     type:
-//        { BOOL | INT64 | FLOAT64 | STRING( length ) | BYTES( length ) | DATE | TIMESTAMP }
+//        { BOOL | INT64 | FLOAT64 | STRING( length ) | BYTES( length ) | DATE | TIMESTAMP | NUMERIC }
 type Type struct {
 	Name string
 	// Len encodes the following Spanner DDL definition:


Test patch:
diff --git a/dynamodb/schema_test.go b/dynamodb/schema_test.go
new file mode 100644
index 0000000..aeeb4ef
--- /dev/null
+++ b/dynamodb/schema_test.go
@@ -0,0 +1,508 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package dynamodb
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/aws/aws-sdk-go/service/dynamodb"
+	"github.com/cloudspannerecosystem/harbourbridge/internal"
+	"github.com/cloudspannerecosystem/harbourbridge/schema"
+	"github.com/cloudspannerecosystem/harbourbridge/spanner/ddl"
+	"github.com/stretchr/testify/assert"
+)
+
+type mockDynamoClient struct {
+	listTableCallCount     int
+	listTableOutputs       []dynamodb.ListTablesOutput
+	describeTableCallCount int
+	describeTableOutputs   []dynamodb.DescribeTableOutput
+	scanCallCount          int
+	scanOutputs            []dynamodb.ScanOutput
+}
+
+func (m *mockDynamoClient) ListTables(input *dynamodb.ListTablesInput) (*dynamodb.ListTablesOutput, error) {
+	if m.listTableCallCount >= len(m.listTableOutputs) {
+		return nil, fmt.Errorf("unexpected call to ListTables: %v", input)
+	}
+	m.listTableCallCount++
+	return &m.listTableOutputs[m.listTableCallCount-1], nil
+}
+
+func (m *mockDynamoClient) DescribeTable(input *dynamodb.DescribeTableInput) (*dynamodb.DescribeTableOutput, error) {
+	if m.describeTableCallCount >= len(m.describeTableOutputs) {
+		return nil, fmt.Errorf("unexpected call to DescribeTable: %v", input)
+	}
+	m.describeTableCallCount++
+	return &m.describeTableOutputs[m.describeTableCallCount-1], nil
+}
+
+func (m *mockDynamoClient) Scan(input *dynamodb.ScanInput) (*dynamodb.ScanOutput, error) {
+	if m.scanCallCount >= len(m.scanOutputs) {
+		return nil, fmt.Errorf("unexpected call to Scan: %v", input)
+	}
+	m.scanCallCount++
+	return &m.scanOutputs[m.scanCallCount-1], nil
+}
+
+func TestProcessSchema(t *testing.T) {
+	tableNameA := "test_a"
+	tableNameB := "test_b"
+	attrNameA := "a"
+	attrNameB := "b"
+	hashKeyType := "HASH"
+	sortKeyType := "RANGE"
+	str := "str"
+	strA := "str-1"
+	strB := "str-2"
+	strC := "str-3"
+	strD := "str-4"
+	numStr := "10"
+
+	listTableOutputs := []dynamodb.ListTablesOutput{
+		{TableNames: []*string{&tableNameA}, LastEvaluatedTableName: &tableNameA},
+		{TableNames: []*string{&tableNameB}},
+	}
+	describeTableOutputs := []dynamodb.DescribeTableOutput{
+		{
+			Table: &dynamodb.TableDescription{
+				TableName: &tableNameA,
+				KeySchema: []*dynamodb.KeySchemaElement{
+					{AttributeName: &attrNameA, KeyType: &hashKeyType},
+				},
+			},
+		},
+		{
+			Table: &dynamodb.TableDescription{
+				TableName: &tableNameB,
+				KeySchema: []*dynamodb.KeySchemaElement{
+					{AttributeName: &attrNameA, KeyType: &hashKeyType},
+					{AttributeName: &attrNameB, KeyType: &sortKeyType},
+				},
+			},
+		},
+	}
+	scanOutputs := []dynamodb.ScanOutput{
+		{
+			Items: []map[string]*dynamodb.AttributeValue{
+				{
+					"a": &dynamodb.AttributeValue{S: &strA},
+					"b": &dynamodb.AttributeValue{S: &str},
+				},
+				{
+					"a": &dynamodb.AttributeValue{S: &strB},
+					"b": &dynamodb.AttributeValue{S: &str},
+				},
+			},
+			LastEvaluatedKey: map[string]*dynamodb.AttributeValue{
+				"a": {S: &strB},
+			},
+		},
+		{
+			Items: []map[string]*dynamodb.AttributeValue{
+				{
+					"a": &dynamodb.AttributeValue{S: &strC},
+					"b": &dynamodb.AttributeValue{N: &numStr},
+				},
+				{
+					"a": &dynamodb.AttributeValue{S: &strD},
+				},
+			},
+		},
+		{
+			Items: []map[string]*dynamodb.AttributeValue{
+				{
+					"a": &dynamodb.AttributeValue{S: &strA},
+					"b": &dynamodb.AttributeValue{S: &str},
+					"c": &dynamodb.AttributeValue{S: &strC},
+				},
+				{
+					"a": &dynamodb.AttributeValue{S: &strB},
+					"b": &dynamodb.AttributeValue{S: &str},
+					"d": &dynamodb.AttributeValue{S: &strD},
+				},
+			},
+		},
+	}
+
+	client := &mockDynamoClient{
+		listTableOutputs:     listTableOutputs,
+		describeTableOutputs: describeTableOutputs,
+		scanOutputs:          scanOutputs,
+	}
+	tables := []string{}
+	sampleSize := int64(10000)
+
+	conv := internal.MakeConv()
+	err := ProcessSchema(conv, client, tables, sampleSize)
+
+	assert.Nil(t, err)
+	expectedSchema := map[string]ddl.CreateTable{
+		"test_a": {
+			Name:     "test_a",
+			ColNames: []string{"a", "b"},
+			ColDefs: map[string]ddl.ColumnDef{
+				"a": {Name: "a", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength}, NotNull: true},
+				"b": {Name: "b", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength}},
+			},
+			Pks: []ddl.IndexKey{{Col: "a"}},
+		},
+		"test_b": {
+			Name:     "test_b",
+			ColNames: []string{"a", "b", "c", "d"},
+			ColDefs: map[string]ddl.ColumnDef{
+				"a": {Name: "a", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength}, NotNull: true},
+				"b": {Name: "b", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength}, NotNull: true},
+				"c": {Name: "c", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength}},
+				"d": {Name: "d", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength}},
+			},
+			Pks: []ddl.IndexKey{{Col: "a"}, {Col: "b"}},
+		},
+	}
+	assert.Equal(t, expectedSchema, stripSchemaComments(conv.SpSchema))
+	assert.Equal(t, int64(0), conv.Unexpecteds())
+}
+
+func TestProcessSchema_FullDataTypes(t *testing.T) {
+	tableNameA := "test_a"
+	attrNameA := "a"
+	attrNameB := "b"
+	hashKeyType := "HASH"
+	sortKeyType := "RANGE"
+
+	str := "str"
+	numStr := "1234.56789"
+	invalidNumStr := "199999999999999999999999999999.999999999"
+	boolVal := true
+	binaryVal := []byte("ABC")
+	listVal := []*dynamodb.AttributeValue{
+		{S: &str},
+		{N: &numStr},
+	}
+	mapVal := map[string]*dynamodb.AttributeValue{
+		"list": {L: listVal},
+	}
+
+	listTableOutputs := []dynamodb.ListTablesOutput{
+		{TableNames: []*string{&tableNameA}},
+	}
+	describeTableOutputs := []dynamodb.DescribeTableOutput{
+		{
+			Table: &dynamodb.TableDescription{
+				TableName: &tableNameA,
+				KeySchema: []*dynamodb.KeySchemaElement{
+					{AttributeName: &attrNameA, KeyType: &hashKeyType},
+					{AttributeName: &attrNameB, KeyType: &sortKeyType},
+				},
+			},
+		},
+	}
+	scanOutputs := []dynamodb.ScanOutput{
+		{
+			Items: []map[string]*dynamodb.AttributeValue{
+				{
+					"a": &dynamodb.AttributeValue{S: &str},
+					"b": &dynamodb.AttributeValue{N: &numStr},
+					"c": &dynamodb.AttributeValue{N: &invalidNumStr},
+					"d": &dynamodb.AttributeValue{BOOL: &boolVal},
+					"e": &dynamodb.AttributeValue{B: binaryVal},
+					"f": &dynamodb.AttributeValue{L: listVal},
+					"g": &dynamodb.AttributeValue{M: mapVal},
+					"h": &dynamodb.AttributeValue{SS: []*string{&str}},
+					"i": &dynamodb.AttributeValue{BS: [][]byte{binaryVal}},
+					"j": &dynamodb.AttributeValue{NS: []*string{&numStr}},
+					"k": &dynamodb.AttributeValue{NS: []*string{&invalidNumStr}},
+				},
+				// The following empty row is needed to make all optional
+				// columns nullable.
+				{},
+			},
+		},
+	}
+
+	client := &mockDynamoClient{
+		listTableOutputs:     listTableOutputs,
+		describeTableOutputs: describeTableOutputs,
+		scanOutputs:          scanOutputs,
+	}
+	tables := []string{}
+	sampleSize := int64(10000)
+
+	conv := internal.MakeConv()
+	err := ProcessSchema(conv, client, tables, sampleSize)
+
+	assert.Nil(t, err)
+	expectedSchema := map[string]ddl.CreateTable{
+		"test_a": {
+			Name:     "test_a",
+			ColNames: []string{"a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k"},
+			ColDefs: map[string]ddl.ColumnDef{
+				"a": {Name: "a", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength}, NotNull: true},
+				"b": {Name: "b", T: ddl.Type{Name: ddl.Numeric}, NotNull: true},
+				"c": {Name: "c", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength}},
+				"d": {Name: "d", T: ddl.Type{Name: ddl.Bool}},
+				"e": {Name: "e", T: ddl.Type{Name: ddl.Bytes, Len: ddl.MaxLength}},
+				"f": {Name: "f", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength}},
+				"g": {Name: "g", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength}},
+				"h": {Name: "h", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength, IsArray: true}},
+				"i": {Name: "i", T: ddl.Type{Name: ddl.Bytes, Len: ddl.MaxLength, IsArray: true}},
+				"j": {Name: "j", T: ddl.Type{Name: ddl.Numeric, IsArray: true}},
+				"k": {Name: "k", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength, IsArray: true}},
+			},
+			Pks: []ddl.IndexKey{{Col: "a"}, {Col: "b"}},
+		},
+	}
+	assert.Equal(t, expectedSchema, stripSchemaComments(conv.SpSchema))
+	assert.Equal(t, int64(0), conv.Unexpecteds())
+}
+
+func TestInferDataTypes(t *testing.T) {
+	dySchema := schema.Table{Name: "test"}
+	stats := map[string]map[string]int64{
+		"all_rows_not_null": {
+			"Number": 1000,
+		},
+		"err_row": {
+			"NumberString": 1,
+			"Number":       999,
+		},
+		"err_null_row": {
+			"Number": 999,
+		},
+		"enough_null_row": {
+			"Number": 900,
+		},
+		"not_conflict_row": {
+			"String": 50,
+			"Number": 950,
+		},
+		"conflict_row": {
+			"String": 51,
+			"Number": 949,
+		},
+		"equal_conflict_rows": {
+			"String": 500,
+			"Number": 500,
+		},
+		"not_conflict_row_with_noise": {
+			"String":       40,
+			"Number":       760,
+			"NumberString": 10,
+		},
+		"conflict_row_with_noise": {
+			"String":       41,
+			"Number":       759,
+			"NumberString": 10,
+		},
+		"equal_conflict_row_with_noise": {
+			"String":       400,
+			"Number":       400,
+			"NumberString": 10,
+		},
+		"empty_records": {
+			"String": 0,
+		},
+		"empty_stats": {},
+	}
+	inferDataTypes(stats, 1000, &dySchema)
+	expectColNames := []string{
+		"all_rows_not_null", "err_row", "err_null_row", "enough_null_row",
+		"not_conflict_row", "conflict_row", "equal_conflict_rows",
+		"not_conflict_row_with_noise", "conflict_row_with_noise",
+		"equal_conflict_row_with_noise",
+	}
+	assert.ElementsMatch(t, expectColNames, dySchema.ColNames)
+	assert.Equal(t, map[string]schema.Column{
+		"all_rows_not_null":             {Name: "all_rows_not_null", Type: schema.Type{Name: "Number"}, NotNull: true},
+		"err_row":                       {Name: "err_row", Type: schema.Type{Name: "Number"}, NotNull: true},
+		"err_null_row":                  {Name: "err_null_row", Type: schema.Type{Name: "Number"}, NotNull: true},
+		"enough_null_row":               {Name: "enough_null_row", Type: schema.Type{Name: "Number"}, NotNull: false},
+		"not_conflict_row":              {Name: "not_conflict_row", Type: schema.Type{Name: "Number"}, NotNull: true},
+		"conflict_row":                  {Name: "conflict_row", Type: schema.Type{Name: "String"}, NotNull: true},
+		"equal_conflict_rows":           {Name: "equal_conflict_rows", Type: schema.Type{Name: "String"}, NotNull: true},
+		"not_conflict_row_with_noise":   {Name: "not_conflict_row_with_noise", Type: schema.Type{Name: "Number"}, NotNull: false},
+		"conflict_row_with_noise":       {Name: "conflict_row_with_noise", Type: schema.Type{Name: "String"}, NotNull: false},
+		"equal_conflict_row_with_noise": {Name: "equal_conflict_row_with_noise", Type: schema.Type{Name: "String"}, NotNull: false},
+	}, dySchema.ColDefs)
+}
+
+func TestScanSampleData(t *testing.T) {
+	strA := "str-1"
+	strB := "str-2"
+	numStr := "10"
+	scanOutputs := []dynamodb.ScanOutput{
+		{
+			Items: []map[string]*dynamodb.AttributeValue{
+				{
+					"a": {S: &strA},
+				},
+				{
+					"a": {N: &numStr},
+				},
+			},
+			LastEvaluatedKey: map[string]*dynamodb.AttributeValue{
+				"a": {S: &strA},
+			},
+		},
+		{
+			Items: []map[string]*dynamodb.AttributeValue{
+				{
+					"a": {S: &strB},
+					"b": {N: &numStr},
+				},
+				{
+					// This will not be scaned due to the sample size.
+					"a": {N: &numStr},
+					"b": {S: &strB},
+				},
+			},
+		},
+	}
+
+	client := &mockDynamoClient{
+		scanOutputs: scanOutputs,
+	}
+
+	stats, _, err := scanSampleData(client, 3, "test")
+	assert.Nil(t, err)
+
+	expectedStats := map[string]map[string]int64{
+		"a": {
+			typeString: 2,
+			typeNumber: 1,
+		},
+		"b": {
+			typeNumber: 1,
+		},
+	}
+	assert.Equal(t, expectedStats, stats)
+}
+
+func TestParseIndexes(t *testing.T) {
+	tableName := "test"
+	attrNameA := "a"
+	attrNameB := "b"
+	attrNameC := "c"
+	hashKeyType := "HASH"
+	sortKeyType := "RANGE"
+	indexName := "secondary_index_c"
+	describeTableOutputs := []dynamodb.DescribeTableOutput{
+		{
+			Table: &dynamodb.TableDescription{
+				TableName: &tableName,
+				KeySchema: []*dynamodb.KeySchemaElement{
+					{AttributeName: &attrNameA, KeyType: &hashKeyType},
+					{AttributeName: &attrNameB, KeyType: &sortKeyType},
+				},
+				GlobalSecondaryIndexes: []*dynamodb.GlobalSecondaryIndexDescription{
+					{
+						IndexName: &indexName,
+						KeySchema: []*dynamodb.KeySchemaElement{
+							{AttributeName: &attrNameC, KeyType: &hashKeyType},
+						},
+					},
+				},
+			},
+		},
+	}
+
+	client := &mockDynamoClient{
+		describeTableOutputs: describeTableOutputs,
+	}
+
+	dySchema := schema.Table{Name: "test"}
+
+	err := analyzeMetadata(client, &dySchema)
+	assert.Nil(t, err)
+
+	pKeys := []schema.Key{{Column: "a"}, {Column: "b"}}
+	assert.Equal(t, pKeys, dySchema.PrimaryKeys)
+	secIndexes := []schema.Index{{Name: "secondary_index_c", Keys: []schema.Key{{Column: "c"}}}}
+	assert.Equal(t, secIndexes, dySchema.Indexes)
+}
+
+func TestListTables(t *testing.T) {
+	tableNameA := "table-a"
+	tableNameB := "table-b"
+
+	listTableOutputs := []dynamodb.ListTablesOutput{
+		{TableNames: []*string{&tableNameA}, LastEvaluatedTableName: &tableNameA},
+		{TableNames: []*string{&tableNameB}},
+	}
+
+	client := &mockDynamoClient{
+		listTableOutputs: listTableOutputs,
+	}
+
+	tables, err := listTables(client)
+	assert.Nil(t, err)
+	assert.Equal(t, []string{"table-a", "table-b"}, tables)
+}
+
+func stripSchemaComments(spSchema map[string]ddl.CreateTable) map[string]ddl.CreateTable {
+	for t, ct := range spSchema {
+		for c, cd := range ct.ColDefs {
+			cd.Comment = ""
+			ct.ColDefs[c] = cd
+		}
+		ct.Comment = ""
+		spSchema[t] = ct
+	}
+	return spSchema
+}
+
+func TestIncTypeCount_Numeric(t *testing.T) {
+	for _, test := range []struct {
+		desc     string
+		in       string
+		wantType string
+	}{
+		{desc: "a positive case", in: "1234.56789", wantType: typeNumber},
+		{desc: "a negative case", in: "-1234.56789", wantType: typeNumber},
+		{desc: "min of numeric", in: "-99999999999999999999999999999.999999999", wantType: typeNumber},
+		{desc: "max of numeric", in: "99999999999999999999999999999.999999999", wantType: typeNumber},
+		{desc: "larger precision", in: "199999999999999999999999999999.999999999", wantType: typeNumberString},
+		{desc: "larger scale", in: "99999999999999999999999999999.9999999991", wantType: typeNumberString},
+		{desc: "smaller precision", in: "-199999999999999999999999999999.999999999", wantType: typeNumberString},
+		{desc: "smaller scale", in: "-99999999999999999999999999999.9999999991", wantType: typeNumberString},
+	} {
+		s := make(map[string]int64)
+		attr := &dynamodb.AttributeValue{N: &test.in}
+		incTypeCount("Revenue", attr, s)
+		assert.Equal(t, s[test.wantType], int64(1))
+	}
+}
+
+func TestIncTypeCount_NumericArray(t *testing.T) {
+	num1 := "1234.56789"
+	num2 := "-1234.56789"
+	num3 := "199999999999999999999999999999.999999999"
+	num4 := "-199999999999999999999999999999.999999999"
+	for _, test := range []struct {
+		desc     string
+		in       []*string
+		wantType string
+	}{
+		{desc: "all valid", in: []*string{&num1, &num2}, wantType: typeNumberSet},
+		{desc: "not valid", in: []*string{&num3, &num4}, wantType: typeNumberStringSet},
+		{desc: "one of the elments is invalid", in: []*string{&num1, &num2, &num3}, wantType: typeNumberStringSet},
+	} {
+		s := make(map[string]int64)
+		attr := &dynamodb.AttributeValue{NS: test.in}
+		incTypeCount("Revenue", attr, s)
+		assert.Equal(t, s[test.wantType], int64(1))
+	}
+}
diff --git a/dynamodb/toddl_test.go b/dynamodb/toddl_test.go
new file mode 100644
index 0000000..2c99646
--- /dev/null
+++ b/dynamodb/toddl_test.go
@@ -0,0 +1,79 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package dynamodb
+
+import (
+	"testing"
+
+	"github.com/cloudspannerecosystem/harbourbridge/internal"
+	"github.com/cloudspannerecosystem/harbourbridge/schema"
+	"github.com/cloudspannerecosystem/harbourbridge/spanner/ddl"
+	"github.com/stretchr/testify/assert"
+)
+
+func TestToSpannerType(t *testing.T) {
+	conv := internal.MakeConv()
+	conv.SetSchemaMode()
+	name := "test"
+	srcSchema := schema.Table{
+		Name:     name,
+		ColNames: []string{"a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k"},
+		ColDefs: map[string]schema.Column{
+			"a": {Name: "a", Type: schema.Type{Name: typeString}},
+			"b": {Name: "b", Type: schema.Type{Name: typeNumber}},
+			"c": {Name: "c", Type: schema.Type{Name: typeNumberString}},
+			"d": {Name: "d", Type: schema.Type{Name: typeBool}},
+			"e": {Name: "e", Type: schema.Type{Name: typeBinary}},
+			"f": {Name: "f", Type: schema.Type{Name: typeList}},
+			"g": {Name: "g", Type: schema.Type{Name: typeMap}},
+			"h": {Name: "h", Type: schema.Type{Name: typeStringSet}},
+			"i": {Name: "i", Type: schema.Type{Name: typeBinarySet}},
+			"j": {Name: "j", Type: schema.Type{Name: typeNumberSet}},
+			"k": {Name: "k", Type: schema.Type{Name: typeNumberStringSet}},
+		},
+		PrimaryKeys: []schema.Key{{Column: "a"}, {Column: "b"}}}
+	conv.SrcSchema[name] = srcSchema
+	assert.Nil(t, schemaToDDL(conv))
+	actual := conv.SpSchema[name]
+	dropComments(&actual) // Don't test comment.
+	expected := ddl.CreateTable{
+		Name:     name,
+		ColNames: []string{"a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k"},
+		ColDefs: map[string]ddl.ColumnDef{
+			"a": {Name: "a", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength}},
+			"b": {Name: "b", T: ddl.Type{Name: ddl.Numeric}},
+			"c": {Name: "c", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength}},
+			"d": {Name: "d", T: ddl.Type{Name: ddl.Bool}},
+			"e": {Name: "e", T: ddl.Type{Name: ddl.Bytes, Len: ddl.MaxLength}},
+			"f": {Name: "f", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength}},
+			"g": {Name: "g", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength}},
+			"h": {Name: "h", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength, IsArray: true}},
+			"i": {Name: "i", T: ddl.Type{Name: ddl.Bytes, Len: ddl.MaxLength, IsArray: true}},
+			"j": {Name: "j", T: ddl.Type{Name: ddl.Numeric, IsArray: true}},
+			"k": {Name: "k", T: ddl.Type{Name: ddl.String, Len: ddl.MaxLength, IsArray: true}},
+		},
+		Pks: []ddl.IndexKey{{Col: "a"}, {Col: "b"}},
+	}
+	assert.Equal(t, expected, actual)
+}
+
+func dropComments(t *ddl.CreateTable) {
+	t.Comment = ""
+	for _, c := range t.ColNames {
+		cd := t.ColDefs[c]
+		cd.Comment = ""
+		t.ColDefs[c] = cd
+	}
+}
diff --git a/mysql/report_test.go b/mysql/report_test.go
index 574b508..c4c4abc 100644
--- a/mysql/report_test.go
+++ b/mysql/report_test.go
@@ -52,6 +52,7 @@ func TestReport(t *testing.T) {
 	conv := internal.MakeConv()
 	conv.SetSchemaMode()
 	ProcessMySQLDump(conv, internal.NewReader(bufio.NewReader(strings.NewReader(s)), nil))
+	conv.SetDataMode()
 	conv.Stats.Rows = map[string]int64{"bad_schema": 1000, "no_pk": 5000}
 	conv.Stats.GoodRows = map[string]int64{"bad_schema": 990, "no_pk": 3000}
 	conv.Stats.BadRows = map[string]int64{"bad_schema": 10, "no_pk": 2000}
@@ -132,7 +133,7 @@ Schema conversion: POOR (many columns did not map cleanly).
 Data conversion: NONE (no data rows found).
 
 Warning
-1) Column 'a' uses foreign keys which Spanner does not support.
+1) Column 'a' uses foreign keys which HarbourBridge does not support yet.
 
 ----------------------------
 Table no_pk
diff --git a/postgres/pgdump_test.go b/postgres/pgdump_test.go
index aa03f80..a5afc41 100644
--- a/postgres/pgdump_test.go
+++ b/postgres/pgdump_test.go
@@ -290,6 +290,14 @@ func TestProcessPgDump(t *testing.T) {
 			expectedData: []spannerData{
 				spannerData{table: "test", cols: []string{"a", "b", "n"}, vals: []interface{}{"a42", "b6", int64(2)}}},
 		},
+		{
+			name: "INSERT with no cols",
+			input: "CREATE TABLE test (a text NOT NULL, b text NOT NULL, n bigint);\n" +
+				"ALTER TABLE ONLY test ADD CONSTRAINT test_pkey PRIMARY KEY (a, b);" +
+				"INSERT INTO test VALUES ('a42', 'b6', 2);",
+			expectedData: []spannerData{
+				spannerData{table: "test", cols: []string{"a", "b", "n"}, vals: []interface{}{"a42", "b6", int64(2)}}},
+		},
 		{
 			name: "INSERT with no primary key",
 			input: "CREATE TABLE test (a text NOT NULL, b text NOT NULL, n bigint);\n" +
diff --git a/postgres/report_test.go b/postgres/report_test.go
index ae16fb6..c818ef0 100644
--- a/postgres/report_test.go
+++ b/postgres/report_test.go
@@ -48,6 +48,7 @@ func TestReport(t *testing.T) {
 	conv := internal.MakeConv()
 	conv.SetSchemaMode()
 	ProcessPgDump(conv, internal.NewReader(bufio.NewReader(strings.NewReader(s)), nil))
+	conv.SetDataMode()
 	conv.Stats.Rows = map[string]int64{"bad_schema": 1000, "no_pk": 5000}
 	conv.Stats.GoodRows = map[string]int64{"bad_schema": 990, "no_pk": 3000}
 	conv.Stats.BadRows = map[string]int64{"bad_schema": 10, "no_pk": 2000}
@@ -134,7 +135,7 @@ Schema conversion: POOR (many columns did not map cleanly).
 Data conversion: NONE (no data rows found).
 
 Warning
-1) Column 'a' uses foreign keys which Spanner does not support.
+1) Column 'a' uses foreign keys which HarbourBridge does not support yet.
 
 ----------------------------
 Table no_pk