Repository: ticdat/ticdat (language: python)

Issue:
Cleaner json 
Partially addresses issue #146 by making "list of lists" the standard json written file for `PanDatFactory`.
Also cleaned up a double warning.
TicDatFactory will find derived foreign keys, but PanDatFactory will not
I think it was probably unwise to put the `findderivedforeignkey` logic in `TicDatFactory`.

I think the most responsible thing would be to make it consistent for both. To avoid backwards compatibility issues, we could just make it so that by default `PanDatFactory` doesn't derive Foreign Keys, and by default, `TicDatFactory` does. But you could have a flag that handles the characteristic similarly in both places.
Overly aggressive string munging when writing to parameters table
See `_infinity_flag_write_cell`
TicDatFactory/PanDatFactory clone should not carry over advanced row predicates
Specifically, if cloning to something other than the same  class type, then the predicate_kwargs_maker won't work. So don't clone over the row predicates that actually use a predicate_kwargs_maker.


Discussion / hints:

Fix patch:
diff --git a/examples/README.md b/examples/README.md
new file mode 100644
index 00000000..495025cc
--- /dev/null
+++ b/examples/README.md
@@ -0,0 +1,47 @@
+## ticdat examples
+
+The examples files here are meant to demonstrate `ticdat` functionality, and to 
+illustrate the principles of 
+[Tidy, Tested, Safe](https://github.com/ticdat/tidy_tested_safe/wiki/What-is-Tidy,-Tested,-Safe%3F). 
+As a rule, I strongly recommend you build your engines as properly versioned Python packages, as opposed to 
+the free standing .py files you see here. Any one of these example files could trivially be
+organized into a dedicated repository [consistent](https://docs.python-guide.org/writing/structure/) with Python best 
+practices. For example, see 
+[tts_diet](https://github.com/ticdat/tts_diet), 
+[tts_netflow_a](https://github.com/ticdat/tts_netflow_a) and 
+[tts_netflow_b](https://github.com/ticdat/tts_netflow_b).
+
+When building an engine with `ticdat`, the first decision to make is how to represent the input data
+tables in Python. There are two choices.
+
+ 1. dict-of-dicts. Some people find this convenient for the idioms associated with Mixed Integer Programming. 
+ In this case, use `TicDatFactory`.
+ 1. `pandas` is an incredibly popular and powerful data library. If you want a 
+ `DataFrame` to be the default representation of a table, then use `PanDatFactory`.
+ 
+ Here is a quick guide to examples for each approach.
+ 
+ * `TicDatFactory`. 
+   * See the `diet.py`, `netflow.py`, `fantop.py`, `metrorail.py` and `workforce.py` files
+ in the `gurobipy` directory.
+ * `PanDatFactory`. 
+   * For a MIP example, see `gurobipy/netflow/netflow_pd.py`. 
+   * The `iris/iris.py` example shows `PanDatFactory` connecting through to the widely used `sklearn` package 
+   for machine learning.
+   * The `datetime/simple_datetime_solver.py` engine demonstrates how `PanDatFactory` can validate and munge timestamp
+   data.
+   
+   Bear in mind that you don't need to spend a lot of mental energy choosing between `TicDatFactory` and 
+   `PanDatFactory`. `ticdat` also makes it very easy to convert the data tables between `DataFrame` and  
+   dict-of-dicts. See `TicDatFactory.copy_to_pandas` and `PanDatFactory.copy_to_tic_dat`.
+   
+   The `beginner_ticdat` directory contains the files relevant to the 
+   [beginner ticdat](https://github.com/ticdat/ticdat/wiki/1-Beginner-ticdat-intro) training material.
+   
+   The `expert_section` directory contains a wide variety of additional examples. This is the place to go if you're 
+   curious about other MIP engines, like `docplex` and `pulp`. Since `ticdat` is just a MIP facilitator, 
+   those examples are very similar to `gurobipy`. 
+   
+   The `expert_section/amplpy` directory demonstrates the `ticdat`  convenience functions that can help you
+   use the AMPL programming language within a Python based engine. Unfortunately, such an approach is not currently 
+   recommended due to the immature state of the AMPL licensing technology.
\ No newline at end of file
diff --git a/examples/amplpy/readme.md b/examples/amplpy/readme.md
deleted file mode 100644
index eab382b6..00000000
--- a/examples/amplpy/readme.md
+++ /dev/null
@@ -1,13 +0,0 @@
-Repository for amplpy examples.
-
-These examples use `PanDatFactory` to connect `amplpy` seamlessly with `pandas`. This is our recommended best practice 
-for OR practitioners looking to utilize a modeling language to implement optimization with Python.
-
-Note that these examples validate input data integrity with both AMPL and `ticdat` code. See [here](https://github.com/opalytics/opalytics-ticdat/wiki/ticdat-and-Optimization-Engines#input-data-integrity-checking-with-ticdat-and-ampl) for more details on the merits of this strategy.
-
- * diet - standard diet example. Easiest example.
- * netflow - standard multi-commodity network flow problem. Second easiest example.
- * fantop - fantasy football example. See [here](https://www.linkedin.com/pulse/fantasy-footballers-nerds-too-peter-cacioppi/) for more information. This is a good example of using `pandas` for pre and post solve data munging.
- * metrorail - optimize the number of times you visit the fare kiosk when riding the metro. See [here](https://www.linkedin.com/pulse/miami-metrorail-meets-python-peter-cacioppi/) for more information. This is a good example of constructing a comprehensive solution from a series of sub-problems.
- * workforce - Demonstrates lexicographical optimization for a workforce scheduling problem.
- * simplest_examples - examples shrunk down by removing the data integrity checking functionality. 
diff --git a/examples/expert_section/ampl_original_examples/diet/diet.py b/examples/expert_section/ampl_original_examples/diet/diet.py
deleted file mode 100644
index e0d8d46a..00000000
--- a/examples/expert_section/ampl_original_examples/diet/diet.py
+++ /dev/null
@@ -1,115 +0,0 @@
-#!/usr/bin/python
-
-# Implement core functionality needed to achieve modularity.
-# 1. Define the input data schema
-# 2. Define the output data schema
-# 3. Create a solve function that accepts a data set consistent with the input
-#    schema and (if possible) returns a data set consistent with the output schema.
-#
-# Provides command line interface via ticdat.standard_main
-# For example, typing
-#   python diet_ampl.py -i input_data.xlsx -o solution_data.xlsx
-# will read from a model stored in the file input_data.xlsx and write the solution
-# to solution_data.xlsx.
-#
-
-# diet_ampl.py doesn't require a separate .mod file. Instead, we use amplpy.AMPL
-from amplpy import AMPL
-from ticdat import TicDatFactory, standard_main
-
-# ------------------------ define the input schema --------------------------------
-# There are three input tables, with 4 primary key fields and 4 data fields.
-input_schema = TicDatFactory (
-    categories = [["Name"],["Min Nutrition", "Max Nutrition"]],
-    foods  = [["Name"],["Cost"]],
-    nutrition_quantities = [["Food", "Category"], ["Quantity"]])
-
-# Define the foreign key relationships
-input_schema.add_foreign_key("nutrition_quantities", "foods", ["Food", "Name"])
-input_schema.add_foreign_key("nutrition_quantities", "categories",
-                            ["Category", "Name"])
-
-# Define the data types
-input_schema.set_data_type("categories", "Min Nutrition", min=0, max=float("inf"),
-                           inclusive_min=True, inclusive_max=False)
-input_schema.set_data_type("categories", "Max Nutrition", min=0, max=float("inf"),
-                           inclusive_min=True, inclusive_max=True)
-input_schema.set_data_type("foods", "Cost", min=0, max=float("inf"),
-                           inclusive_min=True, inclusive_max=False)
-input_schema.set_data_type("nutrition_quantities", "Quantity", min=0, max=float("inf"),
-                           inclusive_min=True, inclusive_max=False)
-
-# We also want to insure that Max Nutrition doesn't fall below Min Nutrition
-input_schema.add_data_row_predicate(
-    "categories", predicate_name="Min Max Check",
-    predicate=lambda row : row["Max Nutrition"] >= row["Min Nutrition"])
-
-# The default-default of zero makes sense everywhere except for Max Nutrition
-input_schema.set_default_value("categories", "Max Nutrition", float("inf"))
-# ---------------------------------------------------------------------------------
-
-# ------------------------ define the output schema -------------------------------
-# There are three solution tables, with 3 primary key fields and 3 data fields.
-solution_schema = TicDatFactory(
-    parameters = [["Parameter"],["Value"]],
-    buy_food = [["Food"],["Quantity"]],
-    consume_nutrition = [["Category"],["Quantity"]])
-# ---------------------------------------------------------------------------------
-
-# ------------------------ create a solve function --------------------------------
-def solve(dat):
-    """
-    core solving routine
-    :param dat: a good ticdat for the input_schema
-    :return: a good ticdat for the solution_schema, or None
-    """
-
-    assert input_schema.good_tic_dat_object(dat)
-    assert not input_schema.find_foreign_key_failures(dat)
-    assert not input_schema.find_data_type_failures(dat)
-    assert not input_schema.find_data_row_failures(dat)
-
-    # copy the data over to amplpy.DataFrame objects, renaming the data fields as needed
-    dat = input_schema.copy_to_ampl(dat, field_renamings={("foods", "Cost"): "cost",
-            ("categories", "Min Nutrition"): "n_min", ("categories", "Max Nutrition"): "n_max",
-            ("nutrition_quantities", "Quantity"): "amt"})
-
-    ampl = AMPL()
-    ampl.setOption('solver', 'gurobi')
-    ampl.eval("""
-    set CAT;
-    set FOOD;
-
-    param cost {FOOD} > 0;
-
-    param n_min {CAT} >= 0;
-    param n_max {i in CAT} >= n_min[i];
-
-    param amt {FOOD, CAT} >= 0;
-
-    var Buy {j in FOOD} >= 0;
-    var Consume {i in CAT } >= n_min [i], <= n_max [i];
-
-    minimize Total_Cost:  sum {j in FOOD} cost[j] * Buy[j];
-
-    subject to Diet {i in CAT}:
-       Consume[i] =  sum {j in FOOD} amt[j,i] * Buy[j];
-    """)
-
-    input_schema.set_ampl_data(dat, ampl, {"categories": "CAT", "foods": "FOOD"})
-    ampl.solve()
-
-    if ampl.getValue("solve_result") != "infeasible":
-        sln = solution_schema.copy_from_ampl_variables(
-            {("buy_food", "Quantity"):ampl.getVariable("Buy"),
-            ("consume_nutrition", "Quantity"):ampl.getVariable("Consume")})
-        sln.parameters['Total Cost'] = ampl.getObjective('Total_Cost').value()
-
-        return sln
-# ---------------------------------------------------------------------------------
-
-# ------------------------ provide stand-alone functionality ----------------------
-# when run from the command line, will read/write xls/csv/db/sql/mdb files;
-if __name__ == "__main__":
-    standard_main(input_schema, solution_schema, solve)
-# ---------------------------------------------------------------------------------
\ No newline at end of file
diff --git a/examples/expert_section/ampl_original_examples/diet/diet_sample_data.sql b/examples/expert_section/ampl_original_examples/diet/diet_sample_data.sql
deleted file mode 100644
index 7119f010..00000000
--- a/examples/expert_section/ampl_original_examples/diet/diet_sample_data.sql
+++ /dev/null
@@ -1,49 +0,0 @@
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('ice cream','protein',8);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('ice cream','fat',10);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('fries','sodium',270);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('fries','calories',380);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('hamburger','fat',26);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('macaroni','sodium',930);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('hot dog','sodium',1800);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('chicken','sodium',1190);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('salad','calories',320);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('ice cream','calories',330);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('milk','sodium',125);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('salad','sodium',1230);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('pizza','sodium',820);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('pizza','protein',15);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('pizza','calories',320);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('hamburger','calories',410);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('milk','fat',2.5);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('salad','protein',31);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('milk','protein',8);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('macaroni','fat',10);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('salad','fat',12);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('hot dog','fat',32);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('chicken','fat',10);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('chicken','protein',32);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('fries','protein',4);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('pizza','fat',12);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('milk','calories',100);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('ice cream','sodium',180);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('chicken','calories',420);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('hamburger','sodium',730);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('macaroni','calories',320);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('fries','fat',19);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('hot dog','calories',560);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('hot dog','protein',20);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('macaroni','protein',12);
-INSERT INTO [nutrition_quantities] ([food],[category],[Quantity]) VALUES ('hamburger','protein',24);
-INSERT INTO [foods] ([name],[cost]) VALUES ('hamburger',2.49);
-INSERT INTO [foods] ([name],[cost]) VALUES ('salad',2.49);
-INSERT INTO [foods] ([name],[cost]) VALUES ('hot dog',1.5);
-INSERT INTO [foods] ([name],[cost]) VALUES ('fries',1.89);
-INSERT INTO [foods] ([name],[cost]) VALUES ('macaroni',2.09);
-INSERT INTO [foods] ([name],[cost]) VALUES ('chicken',2.89);
-INSERT INTO [foods] ([name],[cost]) VALUES ('milk',0.89);
-INSERT INTO [foods] ([name],[cost]) VALUES ('ice cream',1.59);
-INSERT INTO [foods] ([name],[cost]) VALUES ('pizza',1.99);
-INSERT INTO [categories] ([name],[min nutrition],[max nutrition]) VALUES ('protein',91,'inf');
-INSERT INTO [categories] ([name],[min nutrition],[max nutrition]) VALUES ('calories',1800,2200);
-INSERT INTO [categories] ([name],[min nutrition],[max nutrition]) VALUES ('fat',0,65);
-INSERT INTO [categories] ([name],[min nutrition],[max nutrition]) VALUES ('sodium',0,1779);
diff --git a/examples/expert_section/ampl_original_examples/metrorail/create_xls_file.py b/examples/expert_section/ampl_original_examples/metrorail/create_xls_file.py
deleted file mode 100644
index 945092e1..00000000
--- a/examples/expert_section/ampl_original_examples/metrorail/create_xls_file.py
+++ /dev/null
@@ -1,17 +0,0 @@
-# Use this file to convert the metrorail_sample_data.json data set
-# to Excel format.
-#
-#    python create_xls_file.py
-#
-# will create a file named Metro Rail Data.xlsx.
-#
-# metrorail.py will produce the same result regardless of whether
-# it is run on metrorail_sample_data.json or Metro Rail Data.xlsx.
-# This result is largely consistent with the heat map result from
-# https://orbythebeach.wordpress.com/2018/03/01/buying-metrorail-tickets-in-miami/
-# with the exception that we find only two infeasible sub-models.
-
-from metrorail import input_schema
-dat = input_schema.json.create_tic_dat("metrorail_sample_data.json")
-input_schema.xls.write_file(dat, "Metro Rail Data.xlsx", allow_overwrite=True,
-                            case_space_sheet_names=True)
\ No newline at end of file
diff --git a/examples/expert_section/ampl_original_examples/metrorail/metrorail.py b/examples/expert_section/ampl_original_examples/metrorail/metrorail.py
deleted file mode 100644
index 9247edb9..00000000
--- a/examples/expert_section/ampl_original_examples/metrorail/metrorail.py
+++ /dev/null
@@ -1,125 +0,0 @@
-#
-# Models Tallys Yunes Metrorail tickets problem.
-# https://orbythebeach.wordpress.com/2018/03/01/buying-metrorail-tickets-in-miami/
-#
-# Implement core functionality needed to achieve modularity.
-# 1. Define the input data schema
-# 2. Define the output data schema
-# 3. Create a solve function that accepts a data set consistent with the input
-#    schema and (if possible) returns a data set consistent with the output schema.
-#
-# Provides command line interface via ticdat.standard_main
-# For example, typing
-#   python metrorail.py -i metrorail_sample_data.json -o metrorail_solution_data.json
-# will read from a model stored in the file metrorail_sample_data.json and write the
-# solution to metrorail_solution_data.json.
-
-# this version of the file uses amplpy and Gurobi
-from amplpy import AMPL
-from ticdat import TicDatFactory, standard_main
-from itertools import product
-
-# ------------------------ define the input schema --------------------------------
-input_schema = TicDatFactory (
-    parameters=[["Parameter"], ["Value"]],
-    load_amounts=[["Amount"],[]],
-    number_of_one_way_trips=[["Number"],[]],
-    amount_leftover=[["Amount"], []])
-
-input_schema.set_data_type("load_amounts", "Amount", min=0, max=float("inf"),
-                           inclusive_min=False, inclusive_max=False)
-
-input_schema.set_data_type("number_of_one_way_trips", "Number", min=0, max=float("inf"),
-                           inclusive_min=False, inclusive_max=False, must_be_int=True)
-
-input_schema.set_data_type("amount_leftover", "Amount", min=0, max=float("inf"),
-                           inclusive_min=True, inclusive_max=False)
-
-
-default_parameters = {"One Way Price": 2.25, "Amount Leftover Constraint": "Upper Bound"}
-def _good_parameter_key_value(key, value):
-    if key == "One Way Price":
-        try:
-            return 0 < value < float("inf")
-        except:
-            return False
-    if key == "Amount Leftover Constraint":
-        return value  in ["Equality", "Upper Bound"]
-
-assert all(_good_parameter_key_value(k,v) for k,v in default_parameters.items())
-
-input_schema.set_data_type("parameters", "Parameter", number_allowed=False,
-                           strings_allowed=default_parameters)
-input_schema.add_data_row_predicate("parameters", predicate_name="Good Parameter Value",
-    predicate=lambda row : _good_parameter_key_value(row["Parameter"], row["Value"]))
-# ---------------------------------------------------------------------------------
-
-
-# ------------------------ define the output schema -------------------------------
-solution_schema = TicDatFactory(
-    load_amount_details=[["Number One Way Trips", "Amount Leftover", "Load Amount"],
-                           ["Number Of Visits"]],
-    load_amount_summary=[["Number One Way Trips", "Amount Leftover"],["Number Of Visits"]])
-# ---------------------------------------------------------------------------------
-
-
-# ------------------------ create a solve function --------------------------------
-def solve(dat):
-    """
-    core solving routine
-    :param dat: a good ticdat for the input_schema
-    :return: a good ticdat for the solution_schema, or None
-    """
-    assert input_schema.good_tic_dat_object(dat)
-    assert not input_schema.find_foreign_key_failures(dat)
-    assert not input_schema.find_data_type_failures(dat)
-    assert not input_schema.find_data_row_failures(dat)
-    # use default parameters, unless they are overridden by user-supplied parameters
-    full_parameters = dict(default_parameters, **{k:v["Value"] for k,v in dat.parameters.items()})
-
-    sln = solution_schema.TicDat() # create an empty solution
-
-    ampl_dat = input_schema.copy_to_ampl(dat, excluded_tables=
-                   set(input_schema.all_tables).difference({"load_amounts"}))
-    # solve a distinct MIP for each pair of (# of one-way-trips, amount leftover)
-    for number_trips, amount_leftover in product(dat.number_of_one_way_trips, dat.amount_leftover):
-
-        ampl = AMPL()
-        ampl.setOption('solver', 'gurobi')
-        ampl.eval("""
-        param amount_leftover_lb >= 0;
-        param amount_leftover_ub >= amount_leftover_lb;
-        param one_way_price >= 0;
-        param number_trips >= 0;
-        set LOAD_AMTS;
-        var Num_Visits {LOAD_AMTS} integer >= 0;
-        var Amt_Leftover >= amount_leftover_lb, <= amount_leftover_ub;
-        minimize Total_Visits:
-           sum {la in LOAD_AMTS} Num_Visits[la];
-        subj to Set_Amt_Leftover:
-           Amt_Leftover = sum {la in LOAD_AMTS} la * Num_Visits[la] - one_way_price * number_trips;""")
-
-        ampl.param['amount_leftover_lb'] = amount_leftover \
-            if full_parameters["Amount Leftover Constraint"] == "Equality" else 0
-        ampl.param['amount_leftover_ub'] = amount_leftover
-        ampl.param['number_trips'] = number_trips
-        ampl.param['one_way_price'] = full_parameters["One Way Price"]
-        input_schema.set_ampl_data(ampl_dat, ampl, {"load_amounts": "LOAD_AMTS"})
-
-        ampl.solve()
-
-        if ampl.getValue("solve_result") != "infeasible":
-            # store the results if and only if the model is feasible
-            for la,x in ampl.getVariable("Num_Visits").getValues().toDict().items():
-                if round(x) > 0:
-                    sln.load_amount_details[number_trips, amount_leftover, la] = round(x)
-                    sln.load_amount_summary[number_trips, amount_leftover]["Number Of Visits"]\
-                       += round(x)
-    return sln
-# ---------------------------------------------------------------------------------
-
-# ------------------------ provide stand-alone functionality ----------------------
-# when run from the command line, will read/write json/xls/csv/db/sql/mdb files
-if __name__ == "__main__":
-    standard_main(input_schema, solution_schema, solve)
-# ---------------------------------------------------------------------------------
diff --git a/examples/expert_section/ampl_original_examples/metrorail/metrorail_sample_data.json b/examples/expert_section/ampl_original_examples/metrorail/metrorail_sample_data.json
deleted file mode 100644
index 890be352..00000000
--- a/examples/expert_section/ampl_original_examples/metrorail/metrorail_sample_data.json
+++ /dev/null
@@ -1,104 +0,0 @@
-{
-  "amount_leftover": [
-    {
-      "Amount": 0.0
-    },
-    {
-      "Amount": 0.25
-    },
-    {
-      "Amount": 2
-    },
-    {
-      "Amount": 3
-    },
-    {
-      "Amount": 4
-    },
-    {
-      "Amount": 1.25
-    },
-    {
-      "Amount": 1.0
-    },
-    {
-      "Amount": 8.5
-    },
-    {
-      "Amount": 1.75
-    },
-    {
-      "Amount": 0.75
-    },
-    {
-      "Amount": 1.5
-    },
-    {
-      "Amount": 0.5
-    }
-  ],
-  "load_amounts": [
-    {
-      "Amount": 2.25
-    },
-    {
-      "Amount": 1
-    },
-    {
-      "Amount": 3
-    },
-    {
-      "Amount": 5
-    },
-    {
-      "Amount": 4.5
-    },
-    {
-      "Amount": 40
-    },
-    {
-      "Amount": 10
-    },
-    {
-      "Amount": 20
-    }
-  ],
-  "number_of_one_way_trips": [
-    {
-      "Number": 2
-    },
-    {
-      "Number": 4
-    },
-    {
-      "Number": 6
-    },
-    {
-      "Number": 8
-    },
-    {
-      "Number": 10
-    },
-    {
-      "Number": 12
-    },
-    {
-      "Number": 14
-    },
-    {
-      "Number": 16
-    },
-    {
-      "Number": 18
-    },
-    {
-      "Number": 20
-    }
-  ],
-  "parameters": [
-    {
-      "Parameter": "Amount Leftover Constraint",
-      "Value": "Equality"
-    }
-  ]
-}
\ No newline at end of file
diff --git a/examples/expert_section/ampl_original_examples/netflow/netflow.py b/examples/expert_section/ampl_original_examples/netflow/netflow.py
deleted file mode 100644
index 5b5951bf..00000000
--- a/examples/expert_section/ampl_original_examples/netflow/netflow.py
+++ /dev/null
@@ -1,114 +0,0 @@
-#
-# Solve a multi-commodity flow problem as python package.
-#
-# Implement core functionality needed to achieve modularity.
-# 1. Define the input data schema
-# 2. Define the output data schema
-# 3. Create a solve function that accepts a data set consistent with the input
-#    schema and (if possible) returns a data set consistent with the output schema.
-#
-# Provides command line interface via ticdat.standard_main
-# For example, typing
-#   python netflow.py -i netflow_sample_data.sql -o netflow_solution.sql
-# will read from the model stored in netflow_sample_data.sql
-# and write the solution to netflow_solution.sql
-
-from ticdat import TicDatFactory, standard_main
-from amplpy import AMPL
-from itertools import product
-
-# ------------------------ define the input schema --------------------------------
-input_schema = TicDatFactory (
-     commodities = [["Name"],[]],
-     nodes  = [["Name"],[]],
-     arcs = [["Source", "Destination"],["Capacity"]],
-     cost = [["Commodity", "Source", "Destination"], ["Cost"]],
-     inflow = [["Commodity", "Node"],["Quantity"]]
-)
-
-# Define the foreign key relationships
-input_schema.add_foreign_key("arcs", "nodes", ['Source', 'Name'])
-input_schema.add_foreign_key("arcs", "nodes", ['Destination', 'Name'])
-input_schema.add_foreign_key("cost", "nodes", ['Source', 'Name'])
-input_schema.add_foreign_key("cost", "nodes", ['Destination', 'Name'])
-input_schema.add_foreign_key("cost", "commodities", ['Commodity', 'Name'])
-input_schema.add_foreign_key("inflow", "commodities", ['Commodity', 'Name'])
-input_schema.add_foreign_key("inflow", "nodes", ['Node', 'Name'])
-
-# Define the data types
-input_schema.set_data_type("arcs", "Capacity", min=0, max=float("inf"),
-                           inclusive_min=True, inclusive_max=True)
-input_schema.set_data_type("cost", "Cost", min=0, max=float("inf"),
-                           inclusive_min=True, inclusive_max=False)
-input_schema.set_data_type("inflow", "Quantity", min=-float("inf"), max=float("inf"),
-                           inclusive_min=False, inclusive_max=False)
-
-# The default-default of zero makes sense everywhere except for Capacity
-input_schema.set_default_value("arcs", "Capacity", float("inf"))
-# ---------------------------------------------------------------------------------
-
-# ------------------------ define the output schema -------------------------------
-solution_schema = TicDatFactory(
-        flow = [["Commodity", "Source", "Destination"], ["Quantity"]],
-        parameters = [["Parameter"],["Value"]])
-# ---------------------------------------------------------------------------------
-
-# ------------------------ solving section-----------------------------------------
-def solve(dat):
-    """
-    core solving routine
-    :param dat: a good ticdat for the input_schema
-    :return: a good ticdat for the solution_schema, or None
-    """
-    assert input_schema.good_tic_dat_object(dat)
-    assert not input_schema.find_foreign_key_failures(dat)
-    assert not input_schema.find_data_type_failures(dat)
-
-    # copy the data over to amplpy.DataFrame objects, renaming the data fields as needed
-    dat = input_schema.copy_to_ampl(dat, field_renamings={("arcs", "Capacity"): "capacity",
-            ("cost", "Cost"): "cost", ("inflow", "Quantity"): "inflow"})
-
-    # for instructional purposes, the following code anticipates extreme sparsity and doesn't generate
-    # conservation of flow records unless they are really needed
-    ampl = AMPL()
-    ampl.setOption('solver', 'gurobi')
-    ampl.eval("""
-    set NODES;
-    set ARCS within {i in NODES, j in NODES: i <> j};
-    set COMMODITIES;
-    param capacity {ARCS} >= 0;
-    set SHIPMENT_OPTIONS within {COMMODITIES,ARCS};
-    param cost {SHIPMENT_OPTIONS} > 0;
-    set INFLOW_INDEX within {COMMODITIES,NODES};
-    param inflow {INFLOW_INDEX};
-    var Flow {SHIPMENT_OPTIONS} >= 0;
-    minimize TotalCost:
-       sum {(h,i,j) in SHIPMENT_OPTIONS} cost[h,i,j] * Flow[h,i,j];
-    subject to Capacity {(i,j) in ARCS}:
-       sum {(h,i,j) in SHIPMENT_OPTIONS} Flow[h,i,j] <= capacity[i,j];
-    subject to Conservation {h in COMMODITIES, j in NODES:
-         card {(h,i,j) in SHIPMENT_OPTIONS} > 0 or
-         card {(h,j,i) in SHIPMENT_OPTIONS} > 0 or
-         (h,j) in INFLOW_INDEX}:
-       sum {(h,i,j) in SHIPMENT_OPTIONS} Flow[h,i,j] +
-       (if (h,j) in INFLOW_INDEX then inflow[h,j]) =
-       sum {(h,j,i) in SHIPMENT_OPTIONS} Flow[h,j,i];
-    """)
-
-    input_schema.set_ampl_data(dat, ampl, {"nodes": "NODES", "arcs": "ARCS",
-                                           "commodities": "COMMODITIES", "cost":"SHIPMENT_OPTIONS",
-                                            "inflow":"INFLOW_INDEX"})
-    ampl.solve()
-
-    if ampl.getValue("solve_result") != "infeasible":
-        sln = solution_schema.copy_from_ampl_variables(
-            {('flow' ,'Quantity'):ampl.getVariable("Flow")})
-        sln.parameters["Total Cost"] = ampl.getObjective('TotalCost').value()
-        return sln
-# ---------------------------------------------------------------------------------
-
-# ------------------------ provide stand-alone functionality ----------------------
-# when run from the command line, will read/write json/xls/csv/db/sql/mdb files
-if __name__ == "__main__":
-    standard_main(input_schema, solution_schema, solve)
-# ---------------------------------------------------------------------------------
\ No newline at end of file
diff --git a/examples/expert_section/ampl_original_examples/netflow/netflow_sample_data.sql b/examples/expert_section/ampl_original_examples/netflow/netflow_sample_data.sql
deleted file mode 100644
index ce61eebc..00000000
--- a/examples/expert_section/ampl_original_examples/netflow/netflow_sample_data.sql
+++ /dev/null
@@ -1,35 +0,0 @@
-INSERT INTO [commodities] ([name]) VALUES ('Pencils');
-INSERT INTO [commodities] ([name]) VALUES ('Pens');
-INSERT INTO [nodes] ([name]) VALUES ('Seattle');
-INSERT INTO [nodes] ([name]) VALUES ('Boston');
-INSERT INTO [nodes] ([name]) VALUES ('New York');
-INSERT INTO [nodes] ([name]) VALUES ('Denver');
-INSERT INTO [nodes] ([name]) VALUES ('Detroit');
-INSERT INTO [cost] ([commodity],[source],[destination],[cost]) VALUES ('Pencils','Denver','Seattle',30.0);
-INSERT INTO [cost] ([commodity],[source],[destination],[cost]) VALUES ('Pens','Denver','Seattle',30.0);
-INSERT INTO [cost] ([commodity],[source],[destination],[cost]) VALUES ('Pencils','Detroit','New York',20.0);
-INSERT INTO [cost] ([commodity],[source],[destination],[cost]) VALUES ('Pens','Detroit','New York',20.0);
-INSERT INTO [cost] ([commodity],[source],[destination],[cost]) VALUES ('Pens','Detroit','Boston',20.0);
-INSERT INTO [cost] ([commodity],[source],[destination],[cost]) VALUES ('Pencils','Detroit','Boston',10.0);
-INSERT INTO [cost] ([commodity],[source],[destination],[cost]) VALUES ('Pens','Detroit','Seattle',80.0);
-INSERT INTO [cost] ([commodity],[source],[destination],[cost]) VALUES ('Pens','Denver','New York',70.0);
-INSERT INTO [cost] ([commodity],[source],[destination],[cost]) VALUES ('Pens','Denver','Boston',60.0);
-INSERT INTO [cost] ([commodity],[source],[destination],[cost]) VALUES ('Pencils','Denver','New York',40.0);
-INSERT INTO [cost] ([commodity],[source],[destination],[cost]) VALUES ('Pencils','Detroit','Seattle',60.0);
-INSERT INTO [cost] ([commodity],[source],[destination],[cost]) VALUES ('Pencils','Denver','Boston',40.0);
-INSERT INTO [arcs] ([source],[destination],[capacity]) VALUES ('Detroit','Seattle',120.0);
-INSERT INTO [arcs] ([source],[destination],[capacity]) VALUES ('Denver','Seattle',120.0);
-INSERT INTO [arcs] ([source],[destination],[capacity]) VALUES ('Detroit','New York',80.0);
-INSERT INTO [arcs] ([source],[destination],[capacity]) VALUES ('Denver','New York',120.0);
-INSERT INTO [arcs] ([source],[destination],[capacity]) VALUES ('Denver','Boston',120.0);
-INSERT INTO [arcs] ([source],[destination],[capacity]) VALUES ('Detroit','Boston',100.0);
-INSERT INTO [inflow] ([commodity],[node],[quantity]) VALUES ('Pens','Boston',-40.0);
-INSERT INTO [inflow] ([commodity],[node],[quantity]) VALUES ('Pens','New York',-30.0);
-INSERT INTO [inflow] ([commodity],[node],[quantity]) VALUES ('Pencils','Detroit',50.0);
-INSERT INTO [inflow] ([commodity],[node],[quantity]) VALUES ('Pencils','Seattle',-10.0);
-INSERT INTO [inflow] ([commodity],[node],[quantity]) VALUES ('Pencils','Denver',60.0);
-INSERT INTO [inflow] ([commodity],[node],[quantity]) VALUES ('Pens','Detroit',60.0);
-INSERT INTO [inflow] ([commodity],[node],[quantity]) VALUES ('Pens','Seattle',-30.0);
-INSERT INTO [inflow] ([commodity],[node],[quantity]) VALUES ('Pens','Denver',40.0);
-INSERT INTO [inflow] ([commodity],[node],[quantity]) VALUES ('Pencils','New York',-50.0);
-INSERT INTO [inflow] ([commodity],[node],[quantity]) VALUES ('Pencils','Boston',-50.0);
diff --git a/examples/expert_section/ampl_original_examples/readme.md b/examples/expert_section/ampl_original_examples/readme.md
deleted file mode 100644
index efb52c85..00000000
--- a/examples/expert_section/ampl_original_examples/readme.md
+++ /dev/null
@@ -1,10 +0,0 @@
-Repository for amplpy ticdat examples that use `TicDatFactory`.
-
-The `PanDatFactory` examples in examples/ampl demonstrate the recommended best practice.
-
-These examples are being temporarily maintained for legacy reasons.
-
- * diet - standard diet example.
- * netflow - standard multicommodity network flow problems 
- * metrorail - optimize the number of times you visit the fare kiosk when riding the metro. See [here](https://www.linkedin.com/pulse/miami-metrorail-meets-python-peter-cacioppi/) for more information.
- * simplest_examples - examples shrunk down by removing the data integrity checking functionality. Industrial engines are much improved by using this functionality, and I don't recommend skipping pre-condition checking for your engines. That said, most Python-MIP example codes don't include pre-condition checking of the input data, and for this reason I made comparable example codes to facilitate apples-to-apples comparisons.
diff --git a/examples/expert_section/ampl_original_examples/simplest_examples/diet.py b/examples/expert_section/ampl_original_examples/simplest_examples/diet.py
deleted file mode 100644
index 5f0505a9..00000000
--- a/examples/expert_section/ampl_original_examples/simplest_examples/diet.py
+++ /dev/null
@@ -1,62 +0,0 @@
-# Simplest diet example using amplpy, Gurobi and ticdat
-
-from amplpy import AMPL
-from ticdat import TicDatFactory, standard_main
-
-input_schema = TicDatFactory (
-    categories = [["Name"],["Min Nutrition", "Max Nutrition"]],
-    foods  = [["Name"],["Cost"]],
-    nutrition_quantities = [["Food", "Category"], ["Quantity"]])
-
-solution_schema = TicDatFactory(
-    parameters = [["Parameter"],["Value"]],
-    buy_food = [["Food"],["Quantity"]],
-    consume_nutrition = [["Category"],["Quantity"]])
-
-def solve(dat):
-
-    assert input_schema.good_tic_dat_object(dat)
-
-    # copy the data over to amplpy.DataFrame objects, renaming the data fields as needed
-    dat = input_schema.copy_to_ampl(dat, field_renamings={("foods", "Cost"): "cost",
-            ("categories", "Min Nutrition"): "n_min", ("categories", "Max Nutrition"): "n_max",
-            ("nutrition_quantities", "Quantity"): "amt"})
-
-    # create and AMPL object and load it with .mod code
-    ampl = AMPL()
-    ampl.setOption('solver', 'gurobi')
-    ampl.eval("""
-    set CAT;
-    set FOOD;
-
-    param cost {FOOD} > 0;
-
-    param n_min {CAT} >= 0;
-    param n_max {i in CAT} >= n_min[i];
-
-    param amt {FOOD, CAT} >= 0;
-
-    var Buy {j in FOOD} >= 0;
-    var Consume {i in CAT } >= n_min [i], <= n_max [i];
-
-    minimize Total_Cost:  sum {j in FOOD} cost[j] * Buy[j];
-
-    subject to Diet {i in CAT}:
-       Consume[i] =  sum {j in FOOD} amt[j,i] * Buy[j];
-    """)
-
-    # set the AMPL object with the amplpy.DataFrame data and solve
-    input_schema.set_ampl_data(dat, ampl, {"categories": "CAT", "foods": "FOOD"})
-    ampl.solve()
-
-    if ampl.getValue("solve_result") != "infeasible":
-        sln = solution_schema.copy_from_ampl_variables(
-            {("buy_food", "Quantity"):ampl.getVariable("Buy"),
-            ("consume_nutrition", "Quantity"):ampl.getVariable("Consume")})
-        sln.parameters['Total Cost'] = ampl.getObjective('Total_Cost').value()
-
-        return sln
-
-# when run from the command line, will read/write xls/csv/db/sql/mdb/json files
-if __name__ == "__main__":
-    standard_main(input_schema, solution_schema, solve)
\ No newline at end of file
diff --git a/examples/expert_section/ampl_original_examples/simplest_examples/metrorail.py b/examples/expert_section/ampl_original_examples/simplest_examples/metrorail.py
deleted file mode 100644
index 8ebb6151..00000000
--- a/examples/expert_section/ampl_original_examples/simplest_examples/metrorail.py
+++ /dev/null
@@ -1,66 +0,0 @@
-# Simplest metrorail example using amplpy, Gurobi and ticdat
-
-from amplpy import AMPL
-from ticdat import TicDatFactory, standard_main, ampl_format
-from itertools import product
-
-input_schema = TicDatFactory (
-    parameters=[["Parameter"], ["Value"]],
-    load_amounts=[["Amount"],[]],
-    number_of_one_way_trips=[["Number"],[]],
-    amount_leftover=[["Amount"], []])
-
-default_parameters = {"One Way Price": 2.25, "Amount Leftover Constraint": "Upper Bound"}
-
-solution_schema = TicDatFactory(
-    load_amount_details=[["Number One Way Trips", "Amount Leftover", "Load Amount"],
-                           ["Number Of Visits"]],
-    load_amount_summary=[["Number One Way Trips", "Amount Leftover"],["Number Of Visits"]])
-
-def solve(dat):
-    assert input_schema.good_tic_dat_object(dat)
-
-    # use default parameters, unless they are overridden by user-supplied parameters
-    full_parameters = dict(default_parameters, **{k:v["Value"] for k,v in dat.parameters.items()})
-
-    sln = solution_schema.TicDat() # create an empty solution
-
-    # copy the load_amounts table to an amplpy.DataFrame object
-    ampl_dat = input_schema.copy_to_ampl(dat, excluded_tables=
-                   set(input_schema.all_tables).difference({"load_amounts"}))
-
-    # solve a distinct MIP for each pair of (# of one-way-trips, amount leftover)
-    for number_trips, amount_leftover in product(dat.number_of_one_way_trips, dat.amount_leftover):
-
-        # create and AMPL object and load it with .mod code
-        ampl = AMPL()
-        ampl.setOption('solver', 'gurobi')
-        # use the ampl_format function for AMPL friendly key-named text substitutions
-        ampl.eval(ampl_format("""
-        set LOAD_AMTS;
-        var Num_Visits {LOAD_AMTS} integer >= 0;
-        var Amt_Leftover >= {{amount_leftover_lb}}, <= {{amount_leftover_ub}};
-        minimize Total_Visits:
-           sum {la in LOAD_AMTS} Num_Visits[la];
-        subj to Set_Amt_Leftover:
-           Amt_Leftover = sum {la in LOAD_AMTS} la * Num_Visits[la] - {{one_way_price}} * {{number_trips}};""",
-            number_trips=number_trips, one_way_price=full_parameters["One Way Price"],
-            amount_leftover_lb=amount_leftover if full_parameters["Amount Leftover Constraint"] == "Equality" else 0,
-            amount_leftover_ub=amount_leftover))
-
-        # set the AMPL object with the amplpy.DataFrame data and solve
-        input_schema.set_ampl_data(ampl_dat, ampl, {"load_amounts": "LOAD_AMTS"})
-        ampl.solve()
-
-        if ampl.getValue("solve_result") != "infeasible":
-            # store the results if and only if the model is feasible
-            for la,x in ampl.getVariable("Num_Visits").getValues().toDict().items():
-                if round(x[0]) > 0:
-                    sln.load_amount_details[number_trips, amount_leftover, la] = round(x[0])
-                    sln.load_amount_summary[number_trips, amount_leftover]["Number Of Visits"]\
-                       += round(x[0])
-    return sln
-
-# when run from the command line, will read/write json/xls/csv/db/sql/mdb files
-if __name__ == "__main__":
-    standard_main(input_schema, solution_schema, solve)
\ No newline at end of file
diff --git a/examples/expert_section/ampl_original_examples/simplest_examples/netflow.py b/examples/expert_section/ampl_original_examples/simplest_examples/netflow.py
deleted file mode 100644
index eae6b151..00000000
--- a/examples/expert_section/ampl_original_examples/simplest_examples/netflow.py
+++ /dev/null
@@ -1,60 +0,0 @@
-# Simplest netflow example using amplpy, Gurobi and ticdat
-
-from ticdat import TicDatFactory, standard_main
-from amplpy import AMPL
-
-input_schema = TicDatFactory (
-     commodities = [["Name"],[]],
-     nodes  = [["Name"],[]],
-     arcs = [["Source", "Destination"],["Capacity"]],
-     cost = [["Commodity", "Source", "Destination"], ["Cost"]],
-     inflow = [["Commodity", "Node"],["Quantity"]]
-)
-
-solution_schema = TicDatFactory(
-        flow = [["Commodity", "Source", "Destination"], ["Quantity"]],
-        parameters = [["Parameter"],["Value"]])
-
-def solve(dat):
-    assert input_schema.good_tic_dat_object(dat)
-
-    # copy the data over to amplpy.DataFrame objects, renaming the data fields as needed
-    dat = input_schema.copy_to_ampl(dat, field_renamings={("arcs", "Capacity"): "capacity",
-            ("cost", "Cost"): "cost", ("inflow", "Quantity"): "inflow"})
-
-    ampl = AMPL()
-    ampl.setOption('solver', 'gurobi')
-    ampl.eval("""
-    set NODES;
-    set ARCS within {i in NODES, j in NODES: i <> j};
-    set COMMODITIES;
-
-    param capacity {ARCS} >= 0;
-    param cost {COMMODITIES,ARCS} > 0;
-    param inflow {COMMODITIES,NODES};
-
-    var Flow {COMMODITIES,ARCS} >= 0;
-
-    minimize TotalCost:
-       sum {h in COMMODITIES, (i,j) in ARCS} cost[h,i,j] * Flow[h,i,j];
-
-    subject to Capacity {(i,j) in ARCS}:
-       sum {h in COMMODITIES} Flow[h,i,j] <= capacity[i,j];
-
-    subject to Conservation {h in COMMODITIES, j in NODES}:
-       sum {(i,j) in ARCS} Flow[h,i,j] + inflow[h,j] = sum {(j,i) in ARCS} Flow[h,j,i];
-    """)
-
-    input_schema.set_ampl_data(dat, ampl, {"nodes": "NODES", "arcs": "ARCS",
-                                           "commodities": "COMMODITIES"})
-    ampl.solve()
-
-    if ampl.getValue("solve_result") != "infeasible":
-        sln = solution_schema.copy_from_ampl_variables(
-            {('flow' ,'Quantity'):ampl.getVariable("Flow")})
-        sln.parameters["Total Cost"] = ampl.getObjective('TotalCost').value()
-        return sln
-
-# when run from the command line, will read/write json/xls/csv/db/sql/mdb files
-if __name__ == "__main__":
-    standard_main(input_schema, solution_schema, solve)
\ No newline at end of file
diff --git a/examples/amplpy/diet/diet.py b/examples/expert_section/amplpy/diet/diet.py
similarity index 100%
rename from examples/amplpy/diet/diet.py
rename to examples/expert_section/amplpy/diet/diet.py
diff --git a/examples/amplpy/diet/diet_dirty_sample_data/Nutrition Quantities.csv b/examples/expert_section/amplpy/diet/diet_dirty_sample_data/Nutrition Quantities.csv
similarity index 100%
rename from examples/amplpy/diet/diet_dirty_sample_data/Nutrition Quantities.csv
rename to examples/expert_section/amplpy/diet/diet_dirty_sample_data/Nutrition Quantities.csv
diff --git a/examples/amplpy/diet/diet_dirty_sample_data/categories.csv b/examples/expert_section/amplpy/diet/diet_dirty_sample_data/categories.csv
similarity index 100%
rename from examples/amplpy/diet/diet_dirty_sample_data/categories.csv
rename to examples/expert_section/amplpy/diet/diet_dirty_sample_data/categories.csv
diff --git a/examples/amplpy/diet/diet_dirty_sample_data/foods.csv b/examples/expert_section/amplpy/diet/diet_dirty_sample_data/foods.csv
similarity index 100%
rename from examples/amplpy/diet/diet_dirty_sample_data/foods.csv
rename to examples/expert_section/amplpy/diet/diet_dirty_sample_data/foods.csv
diff --git a/examples/amplpy/diet/diet_sample_data/foods.csv b/examples/expert_section/amplpy/diet/diet_sample_data/Foods.csv
similarity index 100%
rename from examples/amplpy/diet/diet_sample_data/foods.csv
rename to examples/expert_section/amplpy/diet/diet_sample_data/Foods.csv
diff --git a/examples/amplpy/diet/diet_sample_data/Nutrition Quantities.csv b/examples/expert_section/amplpy/diet/diet_sample_data/Nutrition Quantities.csv
similarity index 100%
rename from examples/amplpy/diet/diet_sample_data/Nutrition Quantities.csv
rename to examples/expert_section/amplpy/diet/diet_sample_data/Nutrition Quantities.csv
diff --git a/examples/amplpy/diet/diet_sample_data/categories.csv b/examples/expert_section/amplpy/diet/diet_sample_data/categories.csv
similarity index 100%
rename from examples/amplpy/diet/diet_sample_data/categories.csv
rename to examples/expert_section/amplpy/diet/diet_sample_data/categories.csv
diff --git a/examples/amplpy/diet/ticdat_and_dirty_data.ipynb b/examples/expert_section/amplpy/diet/ticdat_and_dirty_data.ipynb
similarity index 100%
rename from examples/amplpy/diet/ticdat_and_dirty_data.ipynb
rename to examples/expert_section/amplpy/diet/ticdat_and_dirty_data.ipynb
diff --git a/examples/amplpy/fantop/create_xlsx_file.py b/examples/expert_section/amplpy/fantop/create_xlsx_file.py
similarity index 100%
rename from examples/amplpy/fantop/create_xlsx_file.py
rename to examples/expert_section/amplpy/fantop/create_xlsx_file.py
diff --git a/examples/amplpy/fantop/fantop.py b/examples/expert_section/amplpy/fantop/fantop.py
similarity index 100%
rename from examples/amplpy/fantop/fantop.py
rename to examples/expert_section/amplpy/fantop/fantop.py
diff --git a/examples/amplpy/fantop/fantop_sample_data.json b/examples/expert_section/amplpy/fantop/fantop_sample_data.json
similarity index 100%
rename from examples/amplpy/fantop/fantop_sample_data.json
rename to examples/expert_section/amplpy/fantop/fantop_sample_data.json
diff --git a/examples/amplpy/fantop/fantop_sample_data/my_draft_positions.csv b/examples/expert_section/amplpy/fantop/fantop_sample_data/my_draft_positions.csv
similarity index 100%
rename from examples/amplpy/fantop/fantop_sample_data/my_draft_positions.csv
rename to examples/expert_section/amplpy/fantop/fantop_sample_data/my_draft_positions.csv
diff --git a/examples/amplpy/fantop/fantop_sample_data/parameters.csv b/examples/expert_section/amplpy/fantop/fantop_sample_data/parameters.csv
similarity index 100%
rename from examples/amplpy/fantop/fantop_sample_data/parameters.csv
rename to examples/expert_section/amplpy/fantop/fantop_sample_data/parameters.csv
diff --git a/examples/amplpy/fantop/fantop_sample_data/players.csv b/examples/expert_section/amplpy/fantop/fantop_sample_data/players.csv
similarity index 100%
rename from examples/amplpy/fantop/fantop_sample_data/players.csv
rename to examples/expert_section/amplpy/fantop/fantop_sample_data/players.csv
diff --git a/examples/amplpy/fantop/fantop_sample_data/roster_requirements.csv b/examples/expert_section/amplpy/fantop/fantop_sample_data/roster_requirements.csv
similarity index 100%
rename from examples/amplpy/fantop/fantop_sample_data/roster_requirements.csv
rename to examples/expert_section/amplpy/fantop/fantop_sample_data/roster_requirements.csv
diff --git a/examples/amplpy/fantop/small_dataset.json b/examples/expert_section/amplpy/fantop/small_dataset.json
similarity index 100%
rename from examples/amplpy/fantop/small_dataset.json
rename to examples/expert_section/amplpy/fantop/small_dataset.json
diff --git a/examples/amplpy/metrorail/create_xls_file.py b/examples/expert_section/amplpy/metrorail/create_xls_file.py
similarity index 100%
rename from examples/amplpy/metrorail/create_xls_file.py
rename to examples/expert_section/amplpy/metrorail/create_xls_file.py
diff --git a/examples/amplpy/metrorail/metrorail.py b/examples/expert_section/amplpy/metrorail/metrorail.py
similarity index 100%
rename from examples/amplpy/metrorail/metrorail.py
rename to examples/expert_section/amplpy/metrorail/metrorail.py
diff --git a/examples/amplpy/metrorail/metrorail_sample_data.json b/examples/expert_section/amplpy/metrorail/metrorail_sample_data.json
similarity index 100%
rename from examples/amplpy/metrorail/metrorail_sample_data.json
rename to examples/expert_section/amplpy/metrorail/metrorail_sample_data.json
diff --git a/examples/amplpy/netflow/netflow.py b/examples/expert_section/amplpy/netflow/netflow.py
similarity index 100%
rename from examples/amplpy/netflow/netflow.py
rename to examples/expert_section/amplpy/netflow/netflow.py
diff --git a/examples/amplpy/netflow/netflow_other_data_sources.ipynb b/examples/expert_section/amplpy/netflow/netflow_other_data_sources.ipynb
similarity index 100%
rename from examples/amplpy/netflow/netflow_other_data_sources.ipynb
rename to examples/expert_section/amplpy/netflow/netflow_other_data_sources.ipynb
diff --git a/examples/amplpy/netflow/netflow_sample_data/arcs.csv b/examples/expert_section/amplpy/netflow/netflow_sample_data/arcs.csv
similarity index 100%
rename from examples/amplpy/netflow/netflow_sample_data/arcs.csv
rename to examples/expert_section/amplpy/netflow/netflow_sample_data/arcs.csv
diff --git a/examples/amplpy/netflow/netflow_sample_data/commodities.csv b/examples/expert_section/amplpy/netflow/netflow_sample_data/commodities.csv
similarity index 100%
rename from examples/amplpy/netflow/netflow_sample_data/commodities.csv
rename to examples/expert_section/amplpy/netflow/netflow_sample_data/commodities.csv
diff --git a/examples/amplpy/netflow/netflow_sample_data/cost.csv b/examples/expert_section/amplpy/netflow/netflow_sample_data/cost.csv
similarity index 100%
rename from examples/amplpy/netflow/netflow_sample_data/cost.csv
rename to examples/expert_section/amplpy/netflow/netflow_sample_data/cost.csv
diff --git a/examples/amplpy/netflow/netflow_sample_data/inflow.csv b/examples/expert_section/amplpy/netflow/netflow_sample_data/inflow.csv
similarity index 100%
rename from examples/amplpy/netflow/netflow_sample_data/inflow.csv
rename to examples/expert_section/amplpy/netflow/netflow_sample_data/inflow.csv
diff --git a/examples/amplpy/netflow/netflow_sample_data/nodes.csv b/examples/expert_section/amplpy/netflow/netflow_sample_data/nodes.csv
similarity index 100%
rename from examples/amplpy/netflow/netflow_sample_data/nodes.csv
rename to examples/expert_section/amplpy/netflow/netflow_sample_data/nodes.csv
diff --git a/examples/expert_section/amplpy/readme.md b/examples/expert_section/amplpy/readme.md
new file mode 100644
index 00000000..5aff7975
--- /dev/null
+++ b/examples/expert_section/amplpy/readme.md
@@ -0,0 +1,31 @@
+Repository for `amplpy` examples.
+
+---
+
+**Watch out** 
+
+**tl/dr** *When it comes to Python integration, AMPL is close but not quite there. I don't recommend it*
+
+While the `amplpy` package is itself very professional, as of this writing, AMPL digital rights management is quite
+primitive. It is far below the level of sophistication of what `gurobipy` allows you to do with a `gurobi.lic` file.
+Until AMPL upgrades their digital rights management, you will likely end up using `amplpy` exclusively to write Python 
+code that has serious portability problems due to the limitations of AMPL's licensing technology. 
+Such problems will likely  render your `amplpy`based work incompatible with modern cloud based architectures. 
+
+Feel free to prove me wrong, however. I am only relaying my experience fruitlessly trying to run AMPL code in a world 
+that is increasingly dominated by Python, Docker, and Kubernetes. If the fault is indeed mine, then I'd love to see a 
+public demonstration proving as much.
+
+---
+
+These examples use `PanDatFactory` to connect `amplpy` seamlessly with `pandas`. 
+
+Note that these examples validate input data integrity with both AMPL and `ticdat` code. `ticdat` is used to make data 
+integrity problems apparent to another Python programmer. AMPL is used to simply double check that the `ticdat` 
+checks are comprehensive.  
+ * diet - standard diet example. Easiest example.
+ * netflow - standard multi-commodity network flow problem. Second easiest example.
+ * fantop - fantasy football example. See [here](https://www.linkedin.com/pulse/fantasy-footballers-nerds-too-peter-cacioppi/) for more information. This is a good example of using `pandas` for pre and post solve data munging.
+ * metrorail - optimize the number of times you visit the fare kiosk when riding the metro. See [here](https://www.linkedin.com/pulse/miami-metrorail-meets-python-peter-cacioppi/) for more information. This is a good example of constructing a comprehensive solution from a series of sub-problems.
+ * workforce - Demonstrates lexicographical optimization for a workforce scheduling problem.
+ * simplest_examples - examples shrunk down by removing the data integrity checking functionality. 
diff --git a/examples/amplpy/simplest_examples/diet.py b/examples/expert_section/amplpy/simplest_examples/diet.py
similarity index 100%
rename from examples/amplpy/simplest_examples/diet.py
rename to examples/expert_section/amplpy/simplest_examples/diet.py
diff --git a/examples/amplpy/simplest_examples/netflow.py b/examples/expert_section/amplpy/simplest_examples/netflow.py
similarity index 100%
rename from examples/amplpy/simplest_examples/netflow.py
rename to examples/expert_section/amplpy/simplest_examples/netflow.py
diff --git a/examples/amplpy/simplest_examples/readme.md b/examples/expert_section/amplpy/simplest_examples/readme.md
similarity index 100%
rename from examples/amplpy/simplest_examples/readme.md
rename to examples/expert_section/amplpy/simplest_examples/readme.md
diff --git a/examples/amplpy/workforce/create_xlsx_file.py b/examples/expert_section/amplpy/workforce/create_xlsx_file.py
similarity index 100%
rename from examples/amplpy/workforce/create_xlsx_file.py
rename to examples/expert_section/amplpy/workforce/create_xlsx_file.py
diff --git a/examples/amplpy/workforce/workforce.py b/examples/expert_section/amplpy/workforce/workforce.py
similarity index 100%
rename from examples/amplpy/workforce/workforce.py
rename to examples/expert_section/amplpy/workforce/workforce.py
diff --git a/examples/amplpy/workforce/workforce_sample_data.json b/examples/expert_section/amplpy/workforce/workforce_sample_data.json
similarity index 100%
rename from examples/amplpy/workforce/workforce_sample_data.json
rename to examples/expert_section/amplpy/workforce/workforce_sample_data.json
diff --git a/examples/expert_section/pandas/slim_pandas/netflow.py b/examples/expert_section/pandas/slim_pandas/netflow.py
new file mode 100644
index 00000000..7ea4ab74
--- /dev/null
+++ b/examples/expert_section/pandas/slim_pandas/netflow.py
@@ -0,0 +1,98 @@
+
+try:
+    import gurobipy as gu
+except:
+    gu = None
+from ticdat import PanDatFactory, standard_main, Slicer
+
+# ------------------------ define the input schema --------------------------------
+input_schema = PanDatFactory (
+    commodities=[["Name"], ["Volume"]],
+    nodes  = [["Name"],[]],
+     arcs = [["Source", "Destination"],["Capacity"]],
+     cost = [["Commodity", "Source", "Destination"], ["Cost"]],
+     inflow = [["Commodity", "Node"],["Quantity"]]
+)
+
+# Define the foreign key relationships
+input_schema.add_foreign_key("arcs", "nodes", ['Source', 'Name'])
+input_schema.add_foreign_key("arcs", "nodes", ['Destination', 'Name'])
+input_schema.add_foreign_key("cost", "arcs", [['Source', 'Source'], ['Destination', 'Destination']])
+input_schema.add_foreign_key("cost", "nodes", ['Destination', 'Name'])
+input_schema.add_foreign_key("cost", "commodities", ['Commodity', 'Name'])
+input_schema.add_foreign_key("inflow", "commodities", ['Commodity', 'Name'])
+input_schema.add_foreign_key("inflow", "nodes", ['Node', 'Name'])
+
+# Define the data types
+input_schema.set_data_type("arcs", "Capacity", min=0, max=float("inf"),
+                           inclusive_min=True, inclusive_max=True)
+input_schema.set_data_type("cost", "Cost", min=0, max=float("inf"),
+                           inclusive_min=True, inclusive_max=False)
+input_schema.set_data_type("inflow", "Quantity", min=-float("inf"), max=float("inf"),
+                           inclusive_min=False, inclusive_max=False)
+input_schema.set_data_type("commodities", "Volume", min=0, max=float("inf"),
+                           inclusive_min=False, inclusive_max=False)
+
+# The default-default of zero makes sense everywhere except for Capacity
+input_schema.set_default_value("arcs", "Capacity", float("inf"))
+# ---------------------------------------------------------------------------------
+
+# ------------------------ define the output schema -------------------------------
+solution_schema = PanDatFactory(
+        flow = [["Commodity", "Source", "Destination"], ["Quantity"]],
+        parameters = [["Parameter"],["Value"]])
+# ---------------------------------------------------------------------------------
+
+# ------------------------ solving section-----------------------------------------
+def solve(dat):
+    """
+    core solving routine
+    :param dat: a good ticdat for the dataFactory
+    :return: a good ticdat for the solutionFactory, or None
+    """
+    assert input_schema.good_pan_dat_object(dat)
+    assert not input_schema.find_duplicates(dat)
+    assert not input_schema.find_foreign_key_failures(dat)
+    assert not input_schema.find_data_type_failures(dat)
+
+
+    # Create optimization model
+    mdl = gu.Model('netflow')
+
+    flow = {(h, i, j): mdl.addVar(name=f'flow_{h}_{i}_{j}', obj=cost)
+            for h, i, j, cost in dat.cost.itertuples(index=False)}
+
+    flowslice = Slicer(flow)
+    volume = {k: volume for k, volume in dat.commodities.itertuples(index=False)}
+
+    # Arc Capacity constraints
+    for i, j, capacity in dat.arcs.itertuples(index=False):
+        mdl.addConstr(gu.quicksum(flow[_h, _i, _j] * volume[_h]
+                                  for _h, _i, _j in flowslice.slice('*', i, j))
+                      <= capacity, name=f'cap_{i}_{j}')
+
+    inflow = {(h, j): qty for h, j, qty in dat.inflow.itertuples(index=False) if abs(qty) > 0}
+    # Flow conservation constraints. Constraints are generated only for relevant pairs.
+    # So we generate a conservation of flow constraint if there is negative or positive inflow
+    # quantity, or at least one inbound flow variable, or at least one outbound flow variable.
+    for h, j in set(inflow).union({(h, i) for h, i, j in flow}, {(h, j) for h, i, j in flow}):
+        mdl.addConstr(
+            gu.quicksum(flow[h_i_j] for h_i_j in flowslice.slice(h, '*', j)) +
+            inflow.get((h, j), 0) ==
+            gu.quicksum(flow[h_j_i] for h_j_i in flowslice.slice(h, j, '*')),
+            name=f'node_{h}_{j}')
+
+    # Compute optimal solution
+    mdl.optimize()
+
+    if mdl.status == gu.GRB.status.OPTIMAL:
+        rtn = solution_schema.PanDat(flow=[[h, i, j, var.x] for (h, i, j), var in flow.items() if var.x > 0],
+                                     parameters=[["Total Cost",  mdl.objVal]])
+        return rtn
+# ---------------------------------------------------------------------------------
+
+# ------------------------ provide stand-alone functionality ----------------------
+# when run from the command line, will read/write xls/csv/db/mdb files
+if __name__ == "__main__":
+    standard_main(input_schema, solution_schema, solve)
+# ---------------------------------------------------------------------------------
diff --git a/examples/gurobipy/diet/diet.py b/examples/gurobipy/diet/diet.py
index 23723962..6558768e 100644
--- a/examples/gurobipy/diet/diet.py
+++ b/examples/gurobipy/diet/diet.py
@@ -21,18 +21,18 @@
 # this version of the file uses Gurobi
 
 try: # if you don't have gurobipy installed, the code will still load and then fail on solve
-    import gurobipy as gu
+    import gurobipy as gp
 except:
-    gu = None
+    gp = None
 
 from ticdat import TicDatFactory, standard_main
 
 # ------------------------ define the input schema --------------------------------
 # There are three input tables, with 4 primary key fields and 4 data fields.
 input_schema = TicDatFactory (
-    categories = [["Name"],["Min Nutrition", "Max Nutrition"]],
-    foods  = [["Name"],["Cost"]],
-    nutrition_quantities = [["Food", "Category"], ["Quantity"]])
+    categories=[["Name"], ["Min Nutrition", "Max Nutrition"]],
+    foods=[["Name"], ["Cost"]],
+    nutrition_quantities=[["Food", "Category"], ["Quantity"]])
 
 # Define the foreign key relationships
 input_schema.add_foreign_key("nutrition_quantities", "foods", ["Food", "Name"])
@@ -62,9 +62,9 @@
 # ------------------------ define the output schema -------------------------------
 # There are three solution tables, with 3 primary key fields and 3 data fields.
 solution_schema = TicDatFactory(
-    parameters = [["Parameter"],["Value"]],
-    buy_food = [["Food"],["Quantity"]],
-    consume_nutrition = [["Category"],["Quantity"]])
+    parameters=[["Parameter"], ["Value"]],
+    buy_food=[["Food"], ["Quantity"]],
+    consume_nutrition=[["Category"], ["Quantity"]])
 # ---------------------------------------------------------------------------------
 
 
@@ -80,27 +80,27 @@ def solve(dat):
     assert not input_schema.find_data_type_failures(dat)
     assert not input_schema.find_data_row_failures(dat)
 
-    if gu is None: # even if you don't have gurobipy installed, you can still import this file for other uses
+    if gp is None: # even if you don't have gurobipy installed, you can still import this file for other uses
         print("*****\ngurobipy needs to be installed for this example code to solve!\n*****\n")
-    mdl = gu.Model("diet")
+    mdl = gp.Model("diet")
 
-    nutrition = {c:mdl.addVar(lb=n["Min Nutrition"], ub=n["Max Nutrition"], name=c)
-                for c,n in dat.categories.items()}
+    nutrition = {c: mdl.addVar(lb=n["Min Nutrition"], ub=n["Max Nutrition"], name=c)
+                for c, n in dat.categories.items()}
 
     # Create decision variables for the foods to buy
-    buy = {f:mdl.addVar(name=f) for f in dat.foods}
+    buy = {f: mdl.addVar(name=f) for f in dat.foods}
 
      # Nutrition constraints
     for c in dat.categories:
-        mdl.addConstr(gu.quicksum(dat.nutrition_quantities[f,c]["Quantity"] * buy[f]
-                      for f in dat.foods) == nutrition[c],
-                      name = c)
+        mdl.addConstr(gp.quicksum(dat.nutrition_quantities[f, c]["Quantity"] * buy[f]
+                                  for f in dat.foods) == nutrition[c],
+                      name=c)
 
-    mdl.setObjective(gu.quicksum(buy[f] * c["Cost"] for f,c in dat.foods.items()),
-                     sense=gu.GRB.MINIMIZE)
+    mdl.setObjective(gp.quicksum(buy[f] * c["Cost"] for f, c in dat.foods.items()),
+                     sense=gp.GRB.MINIMIZE)
     mdl.optimize()
 
-    if mdl.status == gu.GRB.OPTIMAL:
+    if mdl.status == gp.GRB.OPTIMAL:
         sln = solution_schema.TicDat()
         for f,x in buy.items():
             if x.x > 0:
@@ -108,7 +108,7 @@ def solve(dat):
         for c,x in nutrition.items():
             sln.consume_nutrition[c] = x.x
         sln.parameters['Total Cost'] = sum(dat.foods[f]["Cost"] * r["Quantity"]
-                                           for f,r in sln.buy_food.items())
+                                           for f, r in sln.buy_food.items())
         return sln
 # ---------------------------------------------------------------------------------
 
diff --git a/examples/gurobipy/diet/ticdat_and_dirty_data.ipynb b/examples/gurobipy/diet/ticdat_and_dirty_data.ipynb
index 12893a68..d382ab80 100644
--- a/examples/gurobipy/diet/ticdat_and_dirty_data.ipynb
+++ b/examples/gurobipy/diet/ticdat_and_dirty_data.ipynb
@@ -14,9 +14,18 @@
     " 1. Foreign key failures, which check that each record of a child table can cross-reference into the appropriate parent table.\n",
     " 1. Data predicate failures. This checks each row for conditions more complex than the data type failure checks. For example, a maximum column can not be allowed to be smaller than the minimum column.\n",
     " \n",
-    "For a `ticdat` app deployed on Enframe, there will be a dedicated subsection of the input tables dedicated to diagnosing data integrity problems. This subsection is populated whenever an app is solved. There is also an integrity \"Action\" that can be launched to look for data integrity problems independently of the solve process.\n",
+    "For a data scientist building optimization engines, `ticdat` provides bulk-query routines via the `TicDatFactory` and `PanDatFactory` classes. The engine builder can customize these routines to look for data integrity problems that are specific to the problem at hand. Another programmer (i.e. a software engineer embedding the optimization engine into a live system) can then call these subroutines to pre-validate the input data prior to running `solve`. \n",
     "\n",
-    "For a data scientist working offline, `ticdat` provides bulk-query routines that can be used from within a notebook. We briefly tour these routines below. Please consult the docstrings for more information regarding their utility."
+    "We briefly tour these routines below. Please consult the docstrings for more information."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Integrity checking with TicDatFactory \n",
+    "\n",
+    "We will first discuss an optimization engine built with `TicDatFactory`. If you wise to organize your code around `pandas.DataFrame` objects, then the `PanDatFactory` examples further down will be more interesting to you."
    ]
   },
   {
@@ -44,7 +53,7 @@
     {
      "data": {
       "text/plain": [
-       "False"
+       "td: {categories: 4, foods: 9, nutrition_quantities: 36}"
       ]
      },
      "execution_count": 2,
@@ -54,6 +63,26 @@
    ],
    "source": [
     "dat = input_schema.csv.create_tic_dat(\"diet_sample_data\")\n",
+    "dat"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "False"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
     "any (_ for _ in [input_schema.csv.find_duplicates(\"diet_sample_data\"),\n",
     "                 input_schema.find_data_type_failures(dat), \n",
     "                 input_schema.find_foreign_key_failures(dat), \n",
@@ -71,7 +100,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 4,
    "metadata": {},
    "outputs": [
     {
@@ -80,7 +109,7 @@
        "{'nutrition_quantities': {('milk', 'fat'): 2}}"
       ]
      },
-     "execution_count": 3,
+     "execution_count": 4,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -98,16 +127,16 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 5,
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "{TableField(table='nutrition_quantities', field='Quantity'): ValuesPks(bad_values=('',), pks=(('macaroni', 'calories'), ('chicken', 'fat')))}"
+       "{TableField(table='nutrition_quantities', field='Quantity'): ValuesPks(bad_values=('',), pks=(('chicken', 'fat'), ('macaroni', 'calories')))}"
       ]
      },
-     "execution_count": 4,
+     "execution_count": 5,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -119,17 +148,17 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 6,
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "{('nutrition_quantities', 'Quantity'): (('macaroni', 'calories'),\n",
-       "  ('chicken', 'fat'))}"
+       "{('nutrition_quantities', 'Quantity'): (('chicken', 'fat'),\n",
+       "  ('macaroni', 'calories'))}"
       ]
      },
-     "execution_count": 5,
+     "execution_count": 6,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -147,20 +176,20 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 7,
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
        "{('nutrition_quantities', 'foods', ('Food', 'Name')): (('pizza',),\n",
-       "  (('pizza', 'protein'),\n",
+       "  (('pizza', 'fat'),\n",
+       "   ('pizza', 'calories'),\n",
        "   ('pizza', 'sodium'),\n",
-       "   ('pizza', 'fat'),\n",
-       "   ('pizza', 'calories')))}"
+       "   ('pizza', 'protein')))}"
       ]
      },
-     "execution_count": 6,
+     "execution_count": 7,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -178,7 +207,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 8,
    "metadata": {},
    "outputs": [
     {
@@ -187,7 +216,7 @@
        "{TablePredicateName(table='categories', predicate_name='Min Max Check'): ('fat',)}"
       ]
      },
-     "execution_count": 7,
+     "execution_count": 8,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -202,6 +231,362 @@
    "source": [
     "Here, `ticdat` is telling us that the \"Min Max Check\" (i.e. the check that `row[\"Max Nutrition\"] >= row[\"Min Nutrition\"]`) failed for the \"fat\" record of the Categories table. This is easily verified by manual inspection of \"categories.csv\". "
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Integrity checking with PanDatFactory \n",
+    "\n",
+    "`PanDatFactory` is a sibling to `TicDatFactory`, geared for developers who prefer input tables organized into `pandas.DataFrame` objects. The syntax for creating a `PanDatFactory` object is nearly identical to `TicDatFactory`. Refer to [`netflow_pd.py`](https://github.com/ticdat/ticdat/blob/develop/examples/gurobipy/netflow/netflow_pd.py) for an example.\n",
+    "\n",
+    "For expediency, I will close create a `PanDatFactory` clone of the `input_schema` object."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "input_schema_pd = input_schema.clone(clone_factory=ticdat.PanDatFactory)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Data ingestion is done similarly as before."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "pd: {categories: 4, foods: 9, nutrition_quantities: 37}"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dat_pd = input_schema_pd.csv.create_pan_dat(\"diet_dirty_sample_data\")\n",
+    "dat_pd"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We will now present the same integrity check results. Because a `DataFrame` can actually store duplicate rows, the duplicate row check is done with the data object itself (as opposed to the directory of csv files)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'nutrition_quantities':     Food Category  Quantity\n",
+       " 36  milk      fat       1.0}"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "input_schema_pd.find_duplicates(dat_pd)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The other checks are performed the same for `PanDatFactory` as you did for `TicDatFactory`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{TableField(table='nutrition_quantities', field='Quantity'):         Food  Category  Quantity\n",
+       " 22   chicken       fat       NaN\n",
+       " 30  macaroni  calories       NaN}"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "input_schema_pd.find_data_type_failures(dat_pd)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{('nutrition_quantities',\n",
+       "  'foods',\n",
+       "  ('Food', 'Name')):      Food  Category  Quantity\n",
+       " 12  pizza    sodium     820.0\n",
+       " 13  pizza   protein      15.0\n",
+       " 14  pizza  calories     320.0\n",
+       " 25  pizza       fat      12.0}"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "input_schema_pd.find_foreign_key_failures(dat_pd, verbosity=\"Low\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{TablePredicateName(table='categories', predicate_name='Min Max Check'):   Name  Min Nutrition  Max Nutrition\n",
+       " 2  fat             70           65.0}"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "input_schema_pd.find_data_row_failures(dat_pd)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For `PanDatFactory`, the results are all dictionaries, with `pandas.DataFrame` objects as values. I'll deep dive into each result to demonstrate this point."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>Food</th>\n",
+       "      <th>Category</th>\n",
+       "      <th>Quantity</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>36</th>\n",
+       "      <td>milk</td>\n",
+       "      <td>fat</td>\n",
+       "      <td>1.0</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "    Food Category  Quantity\n",
+       "36  milk      fat       1.0"
+      ]
+     },
+     "execution_count": 15,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "input_schema_pd.find_duplicates(dat_pd)['nutrition_quantities']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>Food</th>\n",
+       "      <th>Category</th>\n",
+       "      <th>Quantity</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>12</th>\n",
+       "      <td>pizza</td>\n",
+       "      <td>sodium</td>\n",
+       "      <td>820.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>13</th>\n",
+       "      <td>pizza</td>\n",
+       "      <td>protein</td>\n",
+       "      <td>15.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>14</th>\n",
+       "      <td>pizza</td>\n",
+       "      <td>calories</td>\n",
+       "      <td>320.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>25</th>\n",
+       "      <td>pizza</td>\n",
+       "      <td>fat</td>\n",
+       "      <td>12.0</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "     Food  Category  Quantity\n",
+       "12  pizza    sodium     820.0\n",
+       "13  pizza   protein      15.0\n",
+       "14  pizza  calories     320.0\n",
+       "25  pizza       fat      12.0"
+      ]
+     },
+     "execution_count": 16,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "fk_fails = input_schema_pd.find_foreign_key_failures(dat_pd, verbosity=\"Low\")\n",
+    "fk_fails['nutrition_quantities', 'foods', ('Food', 'Name')]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>Name</th>\n",
+       "      <th>Min Nutrition</th>\n",
+       "      <th>Max Nutrition</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>fat</td>\n",
+       "      <td>70</td>\n",
+       "      <td>65.0</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "  Name  Min Nutrition  Max Nutrition\n",
+       "2  fat             70           65.0"
+      ]
+     },
+     "execution_count": 17,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "input_schema_pd.find_data_row_failures(dat_pd)['categories', 'Min Max Check']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
@@ -220,7 +605,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.1"
+   "version": "3.7.0"
   }
  },
  "nbformat": 4,
diff --git a/examples/gurobipy/fantop/fantop.py b/examples/gurobipy/fantop/fantop.py
index c683a27c..f2154a05 100644
--- a/examples/gurobipy/fantop/fantop.py
+++ b/examples/gurobipy/fantop/fantop.py
@@ -13,16 +13,16 @@
 # A user can thus re-optimize for each of his draft picks.
 #
 # Uses the ticdat package to simplify file IO and provide command line functionality.
-# Can read from .csv, Access, Excel or SQLite files. Self validates the input data
+# Can read from .csv, Access, Excel, JSON or SQLite files. Self validates the input data
 # before solving to prevent strange errors or garbage-in, garbage-out problems.
 
 from ticdat import TicDatFactory, standard_main
 
 
 try: # if you don't have gurobipy installed, the code will still load and then fail on solve
-    import gurobipy as gu
+    import gurobipy as gp
 except:
-    gu = None
+    gp = None
 
 # ------------------------ define the input schema --------------------------------
 input_schema = TicDatFactory (
@@ -106,10 +106,10 @@ def solve(dat):
     can_be_drafted_by_me = {player_name for player_name,row in dat.players.items() if
                             row["Draft Status"] != "Drafted By Someone Else"}
 
-    m = gu.Model('fantop')
-    my_starters = {player_name:m.addVar(vtype=gu.GRB.BINARY, name="starter_%s"%player_name)
+    m = gp.Model('fantop')
+    my_starters = {player_name:m.addVar(vtype=gp.GRB.BINARY, name="starter_%s" % player_name)
                   for player_name in can_be_drafted_by_me}
-    my_reserves = {player_name:m.addVar(vtype=gu.GRB.BINARY, name="reserve_%s"%player_name)
+    my_reserves = {player_name:m.addVar(vtype=gp.GRB.BINARY, name="reserve_%s" % player_name)
                   for player_name in can_be_drafted_by_me}
 
 
@@ -122,12 +122,12 @@ def solve(dat):
                         name="cant_draft_twice_%s"%player_name)
 
     for i,draft_position in enumerate(sorted(dat.my_draft_positions)):
-        m.addConstr(gu.quicksum(my_starters[player_name] + my_reserves[player_name]
+        m.addConstr(gp.quicksum(my_starters[player_name] + my_reserves[player_name]
                                 for player_name in can_be_drafted_by_me
                                 if expected_draft_position[player_name] < draft_position) <= i,
                     name = "at_most_%s_can_be_ahead_of_%s"%(i,draft_position))
 
-    my_draft_size = gu.quicksum(my_starters[player_name] + my_reserves[player_name]
+    my_draft_size = gp.quicksum(my_starters[player_name] + my_reserves[player_name]
                                 for player_name in can_be_drafted_by_me)
     m.addConstr(my_draft_size >= len(already_drafted_by_me) + 1,
                 name = "need_to_extend_by_at_least_one")
@@ -136,8 +136,8 @@ def solve(dat):
     for position, row in dat.roster_requirements.items():
         players = {player_name for player_name in can_be_drafted_by_me
                    if dat.players[player_name]["Position"] == position}
-        starters = gu.quicksum(my_starters[player_name] for player_name in players)
-        reserves = gu.quicksum(my_reserves[player_name] for player_name in players)
+        starters = gp.quicksum(my_starters[player_name] for player_name in players)
+        reserves = gp.quicksum(my_reserves[player_name] for player_name in players)
         m.addConstr(starters >= row["Min Num Starters"], name = "min_starters_%s"%position)
         m.addConstr(starters <= row["Max Num Starters"], name = "max_starters_%s"%position)
         m.addConstr(reserves >= row["Min Num Reserve"], name = "min_reserve_%s"%position)
@@ -146,20 +146,20 @@ def solve(dat):
     parameters = input_schema.create_full_parameters_dict(dat)
     flex_players = {player_name for player_name in can_be_drafted_by_me if
                     dat.roster_requirements[dat.players[player_name]["Position"]]["Flex Status"] == "Flex Eligible"}
-    m.addConstr(gu.quicksum(my_starters[player_name] for player_name in flex_players)
+    m.addConstr(gp.quicksum(my_starters[player_name] for player_name in flex_players)
                 <= parameters["Maximum Number of Flex Starters"],
                 name = "max_flex")
 
     starter_weight = parameters["Starter Weight"]
     reserve_weight = parameters["Reserve Weight"]
-    m.setObjective(gu.quicksum(dat.players[player_name]["Expected Points"] *
+    m.setObjective(gp.quicksum(dat.players[player_name]["Expected Points"] *
                                (my_starters[player_name] * starter_weight + my_reserves[player_name] * reserve_weight)
                                for player_name in can_be_drafted_by_me),
-                   sense=gu.GRB.MAXIMIZE)
+                   sense=gp.GRB.MAXIMIZE)
 
     m.optimize()
 
-    if m.status != gu.GRB.OPTIMAL:
+    if m.status != gp.GRB.OPTIMAL:
         print("No draft at all is possible!")
         return
 
diff --git a/examples/gurobipy/metrorail/metrorail.py b/examples/gurobipy/metrorail/metrorail.py
index 7c7a5c3d..64dab6b4 100644
--- a/examples/gurobipy/metrorail/metrorail.py
+++ b/examples/gurobipy/metrorail/metrorail.py
@@ -16,9 +16,9 @@
 # solution to metrorail_solution_data.json.
 
 try: # if you don't have gurobipy installed, the code will still load and then fail on solve
-    import gurobipy as gu
+    import gurobipy as gp
 except:
-    gu = None
+    gp = None
 from ticdat import TicDatFactory, standard_main
 from itertools import product
 
@@ -71,10 +71,10 @@ def solve(dat):
     # solve a distinct MIP for each pair of (# of one-way-trips, amount leftover)
     for number_trips, amount_leftover in product(dat.number_of_one_way_trips, dat.amount_leftover):
 
-        mdl = gu.Model("metrorail")
+        mdl = gp.Model("metrorail")
 
         # Create decision variables
-        number_vists = {la:mdl.addVar(vtype = gu.GRB.INTEGER, name="load_amount_%s"%la)
+        number_vists = {la:mdl.addVar(vtype = gp.GRB.INTEGER, name="load_amount_%s" % la)
                         for la in dat.load_amounts}
         amount_leftover_var = mdl.addVar(name="amount_leftover", lb=0, ub=amount_leftover)
 
@@ -84,22 +84,22 @@ def solve(dat):
         # for left-over is multiple, we will still respect the amount leftover upper bound
         # but will also enforce that the amount leftover is a multiple of the one way price
         if full_parameters["Amount Leftover Constraint"] == "Upper Bound With Leftover Multiple Rule":
-            leftover_multiple = mdl.addVar(vtype = gu.GRB.INTEGER, name="leftover_multiple")
+            leftover_multiple = mdl.addVar(vtype = gp.GRB.INTEGER, name="leftover_multiple")
             mdl.addConstr(amount_leftover_var == full_parameters["One Way Price"] * leftover_multiple,
                           name="set_leftover_multiple")
 
         # add a constraint to set the amount leftover
         mdl.addConstr(amount_leftover_var ==
-                      gu.quicksum(la * number_vists[la] for la in dat.load_amounts) -
+                      gp.quicksum(la * number_vists[la] for la in dat.load_amounts) -
                       full_parameters["One Way Price"] * number_trips,
                       name="set_amount_leftover")
 
 
         # minimize the total number of visits to the ticket office
-        mdl.setObjective(gu.quicksum(number_vists.values()), sense=gu.GRB.MINIMIZE)
+        mdl.setObjective(gp.quicksum(number_vists.values()), sense=gp.GRB.MINIMIZE)
         mdl.optimize()
 
-        if mdl.status in [gu.GRB.OPTIMAL, gu.GRB.SUBOPTIMAL]:
+        if mdl.status in [gp.GRB.OPTIMAL, gp.GRB.SUBOPTIMAL]:
             # store the results if and only if the model is feasible
             for la,x in number_vists.items():
                 if round(x.x) > 0:
diff --git a/examples/gurobipy/netflow/netflow.py b/examples/gurobipy/netflow/netflow.py
index e8d927b1..a0f6d25c 100644
--- a/examples/gurobipy/netflow/netflow.py
+++ b/examples/gurobipy/netflow/netflow.py
@@ -1,5 +1,5 @@
 #
-# Solve a multi-commodity flow problem as python package.
+# Solve a multi-commodity flow problem.
 #
 # Implement core functionality needed to achieve modularity.
 # 1. Define the input data schema
@@ -17,9 +17,9 @@
 # unneeded constraints. See the simplest_examples directory for a simpler version of this model.
 
 try: # if you don't have gurobipy installed, the code will still load and then fail on solve
-    import gurobipy as gu
+    import gurobipy as gp
 except:
-    gu = None
+    gp = None
 from ticdat import TicDatFactory, standard_main, Slicer
 
 # ------------------------ define the input schema --------------------------------
@@ -71,7 +71,7 @@ def solve(dat):
     assert not input_schema.find_foreign_key_failures(dat)
     assert not input_schema.find_data_type_failures(dat)
 
-    mdl = gu.Model("netflow")
+    mdl = gp.Model("netflow")
 
     flow = {(h, i, j): mdl.addVar(name='flow_%s_%s_%s' % (h, i, j))
             for h, i, j in dat.cost if (i,j) in dat.arcs}
@@ -80,7 +80,7 @@ def solve(dat):
 
     # Arc Capacity constraints
     for i,j in dat.arcs:
-        mdl.addConstr(gu.quicksum(flow[_h, _i, _j] * dat.commodities[_h]["Volume"]
+        mdl.addConstr(gp.quicksum(flow[_h, _i, _j] * dat.commodities[_h]["Volume"]
                                   for _h, _i, _j in flowslice.slice('*', i, j))
                       <= dat.arcs[i,j]["Capacity"],
                       name='cap_%s_%s' % (i, j))
@@ -91,17 +91,17 @@ def solve(dat):
     for h,j in set((h,i) for (h,i), v in dat.inflow.items() if abs(v["Quantity"]) > 0)\
                .union({(h,i) for h,i,j in flow}, {(h,j) for h,i,j in flow}):
         mdl.addConstr(
-            gu.quicksum(flow[h_i_j] for h_i_j in flowslice.slice(h,'*',j)) +
+            gp.quicksum(flow[h_i_j] for h_i_j in flowslice.slice(h, '*', j)) +
             dat.inflow.get((h,j), {"Quantity":0})["Quantity"] ==
-            gu.quicksum(flow[h_j_i] for h_j_i in flowslice.slice(h, j, '*')),
+            gp.quicksum(flow[h_j_i] for h_j_i in flowslice.slice(h, j, '*')),
             name='node_%s_%s' % (h, j))
 
-    mdl.setObjective(gu.quicksum(flow * dat.cost[h, i, j]["Cost"]
+    mdl.setObjective(gp.quicksum(flow * dat.cost[h, i, j]["Cost"]
                                  for (h, i, j), flow in flow.items()),
-                     sense=gu.GRB.MINIMIZE)
+                     sense=gp.GRB.MINIMIZE)
     mdl.optimize()
 
-    if mdl.status == gu.GRB.OPTIMAL:
+    if mdl.status == gp.GRB.OPTIMAL:
         rtn = solution_schema.TicDat()
         for (h, i, j), var in flow.items():
             if var.x > 0:
diff --git a/examples/gurobipy/netflow/netflow_other_data_sources.ipynb b/examples/gurobipy/netflow/netflow_other_data_sources.ipynb
index 337c4533..35ee80e3 100644
--- a/examples/gurobipy/netflow/netflow_other_data_sources.ipynb
+++ b/examples/gurobipy/netflow/netflow_other_data_sources.ipynb
@@ -28,29 +28,29 @@
     "# a one column table can just be a simple list \n",
     "nodes = ['Boston', 'Denver', 'Detroit', 'New York',  'Seattle']\n",
     "\n",
-    "cost = [['Pencils', 'Denver', 'Boston', 10.0],\n",
-    "        ['Pencils', 'Denver', 'New York', 10.0],\n",
-    "        ['Pencils', 'Denver', 'Seattle', 7.5],\n",
-    "        ['Pencils', 'Detroit', 'Boston', 2.5],\n",
-    "        ['Pencils', 'Detroit', 'New York', 5.0],\n",
-    "        ['Pencils', 'Detroit', 'Seattle', 15.0],\n",
-    "        ['Pens', 'Denver', 'Boston', 15.0],\n",
-    "        ['Pens', 'Denver', 'New York', 17.5],\n",
-    "        ['Pens', 'Denver', 'Seattle', 7.5],\n",
-    "        ['Pens', 'Detroit', 'Boston', 5.0],\n",
-    "        ['Pens', 'Detroit', 'New York', 5.0],\n",
-    "        ['Pens', 'Detroit', 'Seattle', 20.0]]\n",
+    "cost = [['Pencils', 'Denver', 'Seattle', 30.0],\n",
+    "        ['Pens', 'Denver', 'Seattle', 30.0],\n",
+    "        ['Pencils', 'Detroit', 'New York', 20.0],\n",
+    "        ['Pens', 'Detroit', 'New York', 20.0],\n",
+    "        ['Pens', 'Detroit', 'Boston', 20.0],\n",
+    "        ['Pencils', 'Detroit', 'Boston', 10.0],\n",
+    "        ['Pens', 'Detroit', 'Seattle', 80.0],\n",
+    "        ['Pens', 'Denver', 'New York', 70.0],\n",
+    "        ['Pens', 'Denver', 'Boston', 60.0],\n",
+    "        ['Pencils', 'Denver', 'New York', 40.0],\n",
+    "        ['Pencils', 'Detroit', 'Seattle', 60.0],\n",
+    "        ['Pencils', 'Denver', 'Boston', 40.0]]\n",
     "\n",
-    "inflow = [['Pencils', 'Boston', -200],\n",
-    "          ['Pencils', 'Denver', 240],\n",
-    "          ['Pencils', 'Detroit', 200],\n",
-    "          ['Pencils', 'New York', -200],\n",
-    "          ['Pencils', 'Seattle', -40],\n",
-    "          ['Pens', 'Boston', -160],\n",
-    "          ['Pens', 'Denver', 160],\n",
-    "          ['Pens', 'Detroit', 240],\n",
-    "          ['Pens', 'New York', -120],\n",
-    "          ['Pens', 'Seattle', -120]]"
+    "inflow = [['Pens', 'Boston', -40.0],\n",
+    "          ['Pens', 'New York', -30.0],\n",
+    "          ['Pencils', 'Detroit', 50.0],\n",
+    "          ['Pencils', 'Seattle', -10.0],\n",
+    "          ['Pencils', 'Denver', 60.0],\n",
+    "          ['Pens', 'Detroit', 60.0],\n",
+    "          ['Pens', 'Seattle', -30.0],\n",
+    "          ['Pens', 'Denver', 40.0],\n",
+    "          ['Pencils', 'New York', -50.0],\n",
+    "          ['Pencils', 'Boston', -50.0]]"
    ]
   },
   {
@@ -86,39 +86,45 @@
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
-       "      <th></th>\n",
-       "      <th>Capacity</th>\n",
-       "    </tr>\n",
-       "    <tr>\n",
        "      <th>Source</th>\n",
        "      <th>Destination</th>\n",
-       "      <th></th>\n",
+       "      <th>Capacity</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
-       "      <td rowspan=\"3\" valign=\"top\">Denver</td>\n",
+       "      <th>0</th>\n",
+       "      <td>Denver</td>\n",
        "      <td>Boston</td>\n",
        "      <td>120</td>\n",
        "    </tr>\n",
        "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>Denver</td>\n",
        "      <td>New York</td>\n",
        "      <td>120</td>\n",
        "    </tr>\n",
        "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>Denver</td>\n",
        "      <td>Seattle</td>\n",
        "      <td>120</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <td rowspan=\"3\" valign=\"top\">Detroit</td>\n",
+       "      <th>3</th>\n",
+       "      <td>Detroit</td>\n",
        "      <td>Boston</td>\n",
        "      <td>100</td>\n",
        "    </tr>\n",
        "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>Detroit</td>\n",
        "      <td>New York</td>\n",
        "      <td>80</td>\n",
        "    </tr>\n",
        "    <tr>\n",
+       "      <th>5</th>\n",
+       "      <td>Detroit</td>\n",
        "      <td>Seattle</td>\n",
        "      <td>120</td>\n",
        "    </tr>\n",
@@ -127,14 +133,13 @@
        "</div>"
       ],
       "text/plain": [
-       "                     Capacity\n",
-       "Source  Destination          \n",
-       "Denver  Boston            120\n",
-       "        New York          120\n",
-       "        Seattle           120\n",
-       "Detroit Boston            100\n",
-       "        New York           80\n",
-       "        Seattle           120"
+       "    Source Destination  Capacity\n",
+       "0   Denver      Boston       120\n",
+       "1   Denver    New York       120\n",
+       "2   Denver     Seattle       120\n",
+       "3  Detroit      Boston       100\n",
+       "4  Detroit    New York        80\n",
+       "5  Detroit     Seattle       120"
       ]
      },
      "execution_count": 2,
@@ -147,7 +152,7 @@
     "arcs = DataFrame({\"Source\": [\"Denver\", \"Denver\", \"Denver\", \"Detroit\", \"Detroit\", \"Detroit\",], \n",
     "                 \"Destination\": [\"Boston\", \"New York\", \"Seattle\", \"Boston\", \"New York\", \n",
     "                                 \"Seattle\"], \n",
-    "                 \"Capacity\": [120, 120, 120, 100, 80, 120]}).set_index([\"Source\", \"Destination\"])\n",
+    "                 \"Capacity\": [120, 120, 120, 100, 80, 120]})\n",
     "arcs"
    ]
   },
@@ -155,7 +160,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Next we create a TicDat input data object from the list-of-lists/`DataFrame` representations."
+    "Next we create a `TicDat` input data object from the list-of-lists/`DataFrame` representations."
    ]
   },
   {
@@ -164,8 +169,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from netflow import input_schema, solve, solution_schema\n",
-    "dat = input_schema.TicDat(commodities=commodities, nodes=nodes, cost=cost, arcs=arcs, \n",
+    "import netflow\n",
+    "dat = netflow.input_schema.TicDat(commodities=commodities, nodes=nodes, cost=cost, arcs=arcs, \n",
     "                          inflow=inflow)"
    ]
   },
@@ -185,56 +190,289 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
+      "Using license file /Users/petercacioppi/gurobi.lic\n",
+      "Gurobi Optimizer version 9.1.2 build v9.1.2rc0 (mac64)\n",
+      "Thread count: 4 physical cores, 4 logical processors, using up to 4 threads\n",
       "Optimize a model with 16 rows, 12 columns and 36 nonzeros\n",
+      "Model fingerprint: 0x6b98959e\n",
       "Coefficient statistics:\n",
       "  Matrix range     [2e-01, 1e+00]\n",
-      "  Objective range  [2e+00, 2e+01]\n",
+      "  Objective range  [1e+01, 8e+01]\n",
       "  Bounds range     [0e+00, 0e+00]\n",
-      "  RHS range        [4e+01, 2e+02]\n",
-      "Presolve removed 12 rows and 6 columns\n",
+      "  RHS range        [1e+01, 1e+02]\n",
+      "Presolve removed 16 rows and 12 columns\n",
       "Presolve time: 0.01s\n",
-      "Presolved: 4 rows, 8 columns, 12 nonzeros\n",
-      "\n",
+      "Presolve: All rows and columns removed\n",
       "Iteration    Objective       Primal Inf.    Dual Inf.      Time\n",
-      "       0    2.8823529e+02   1.717647e+02   0.000000e+00      0s\n",
-      "       3    5.6275000e+03   0.000000e+00   0.000000e+00      0s\n",
+      "       0    5.5000000e+03   0.000000e+00   2.000000e+01      0s\n",
+      "Extra simplex iterations after uncrush: 1\n",
+      "       1    5.5000000e+03   0.000000e+00   0.000000e+00      0s\n",
       "\n",
-      "Solved in 3 iterations and 0.02 seconds\n",
-      "Optimal objective  5.627500000e+03\n"
+      "Solved in 1 iterations and 0.02 seconds\n",
+      "Optimal objective  5.500000000e+03\n"
      ]
     }
    ],
    "source": [
-    "sln = solve(dat)"
+    "sln = netflow.solve(dat)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "td: {flow: 7, parameters: 1}"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "sln"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{('Pencils', 'Denver', 'Seattle'): _td:{'Quantity': 10.0},\n",
+       " ('Pens', 'Denver', 'Seattle'): _td:{'Quantity': 30.0},\n",
+       " ('Pens', 'Detroit', 'New York'): _td:{'Quantity': 30.0},\n",
+       " ('Pens', 'Detroit', 'Boston'): _td:{'Quantity': 30.0},\n",
+       " ('Pencils', 'Detroit', 'Boston'): _td:{'Quantity': 50.0},\n",
+       " ('Pens', 'Denver', 'Boston'): _td:{'Quantity': 10.0},\n",
+       " ('Pencils', 'Denver', 'New York'): _td:{'Quantity': 50.0}}"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "sln.flow"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We now create a list of list representation of the solution data object."
+    "We can then easily convert the `sln` object to represent the solution tables as `pandas.DataFrame` objects."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 7,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "pd: {flow: 7, parameters: 1}"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "sln_to_pandas = netflow.solution_schema.copy_to_pandas(sln, reset_index=True)\n",
+    "sln_to_pandas"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>Parameter</th>\n",
+       "      <th>Value</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>Total Cost</td>\n",
+       "      <td>5500.0</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "    Parameter   Value\n",
+       "0  Total Cost  5500.0"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "sln_to_pandas.parameters"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>Commodity</th>\n",
+       "      <th>Source</th>\n",
+       "      <th>Destination</th>\n",
+       "      <th>Quantity</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>Pencils</td>\n",
+       "      <td>Denver</td>\n",
+       "      <td>Seattle</td>\n",
+       "      <td>10.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>Pens</td>\n",
+       "      <td>Denver</td>\n",
+       "      <td>Seattle</td>\n",
+       "      <td>30.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>Pens</td>\n",
+       "      <td>Detroit</td>\n",
+       "      <td>New York</td>\n",
+       "      <td>30.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>Pens</td>\n",
+       "      <td>Detroit</td>\n",
+       "      <td>Boston</td>\n",
+       "      <td>30.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>Pencils</td>\n",
+       "      <td>Detroit</td>\n",
+       "      <td>Boston</td>\n",
+       "      <td>50.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>5</th>\n",
+       "      <td>Pens</td>\n",
+       "      <td>Denver</td>\n",
+       "      <td>Boston</td>\n",
+       "      <td>10.0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>6</th>\n",
+       "      <td>Pencils</td>\n",
+       "      <td>Denver</td>\n",
+       "      <td>New York</td>\n",
+       "      <td>50.0</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "  Commodity   Source Destination  Quantity\n",
+       "0   Pencils   Denver     Seattle      10.0\n",
+       "1      Pens   Denver     Seattle      30.0\n",
+       "2      Pens  Detroit    New York      30.0\n",
+       "3      Pens  Detroit      Boston      30.0\n",
+       "4   Pencils  Detroit      Boston      50.0\n",
+       "5      Pens   Denver      Boston      10.0\n",
+       "6   Pencils   Denver    New York      50.0"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
-    "from ticdat.jsontd import make_json_dict\n",
-    "sln_lists = make_json_dict(solution_schema, sln)"
+    "sln_to_pandas.flow"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Here we demonstrate that `sln_lists` is a dictionary mapping table name to list-of-lists of solution report data."
+    "Or course, to convert from `DataFrame` to list-of-lists is trivial."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "sln_lists = {}\n",
+    "for k in netflow.solution_schema.all_tables:\n",
+    "    df = getattr(sln_to_pandas, k)\n",
+    "    sln_lists[k] = [list(row) for row in df.itertuples(index=False)]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
    "metadata": {},
    "outputs": [
     {
@@ -244,23 +482,21 @@
       "\n",
       "\n",
       "**\n",
-      "Solution Table flow\n",
+      "Solution Table parameters\n",
       "**\n",
-      "[['Pencils', 'Denver', 'Boston', 51.0],\n",
-      " ['Pencils', 'Denver', 'New York', 149.0],\n",
-      " ['Pencils', 'Denver', 'Seattle', 40.0],\n",
-      " ['Pencils', 'Detroit', 'Boston', 149.0],\n",
-      " ['Pencils', 'Detroit', 'New York', 51.0],\n",
-      " ['Pens', 'Denver', 'Boston', 40.0],\n",
-      " ['Pens', 'Denver', 'Seattle', 120.0],\n",
-      " ['Pens', 'Detroit', 'Boston', 120.0],\n",
-      " ['Pens', 'Detroit', 'New York', 120.0]]\n",
+      "[['Total Cost', 5500.0]]\n",
       "\n",
       "\n",
       "**\n",
-      "Solution Table parameters\n",
+      "Solution Table flow\n",
       "**\n",
-      "[['Total Cost', 5627.5]]\n"
+      "[['Pencils', 'Denver', 'Seattle', 10.0],\n",
+      " ['Pens', 'Denver', 'Seattle', 30.0],\n",
+      " ['Pens', 'Detroit', 'New York', 30.0],\n",
+      " ['Pens', 'Detroit', 'Boston', 30.0],\n",
+      " ['Pencils', 'Detroit', 'Boston', 50.0],\n",
+      " ['Pens', 'Denver', 'Boston', 10.0],\n",
+      " ['Pencils', 'Denver', 'New York', 50.0]]\n"
      ]
     }
    ],
@@ -275,12 +511,84 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Here we demonstrate how `DataFrame` objects can be generated from the solution data object."
+    "## Modularity with PanDatFactory\n",
+    "The `netflow` code uses `TicDatFactory` for `input_schema` and `solution_schema`. If you prefer organizing your `solve` code around `DataFrame` objects, then you will likely prefer `PanDatFactory`. This is demonstrated with the `netflow_pd` engine. \n",
+    "\n",
+    "Here, we run through the same steps using `netflow_pd`."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "pd: {arcs: 6, commodities: 2, cost: 12, inflow: 10, nodes: 5}"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import netflow_pd\n",
+    "dat_pd = netflow_pd.input_schema.PanDat(commodities=commodities, nodes=nodes, cost=cost, arcs=arcs, \n",
+    "                                     inflow=inflow)\n",
+    "dat_pd"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Gurobi Optimizer version 9.1.2 build v9.1.2rc0 (mac64)\n",
+      "Thread count: 4 physical cores, 4 logical processors, using up to 4 threads\n",
+      "Optimize a model with 16 rows, 12 columns and 36 nonzeros\n",
+      "Model fingerprint: 0x6b98959e\n",
+      "Coefficient statistics:\n",
+      "  Matrix range     [2e-01, 1e+00]\n",
+      "  Objective range  [1e+01, 8e+01]\n",
+      "  Bounds range     [0e+00, 0e+00]\n",
+      "  RHS range        [1e+01, 1e+02]\n",
+      "Presolve removed 16 rows and 12 columns\n",
+      "Presolve time: 0.01s\n",
+      "Presolve: All rows and columns removed\n",
+      "Iteration    Objective       Primal Inf.    Dual Inf.      Time\n",
+      "       0    5.5000000e+03   0.000000e+00   2.000000e+01      0s\n",
+      "Extra simplex iterations after uncrush: 1\n",
+      "       1    5.5000000e+03   0.000000e+00   0.000000e+00      0s\n",
+      "\n",
+      "Solved in 1 iterations and 0.01 seconds\n",
+      "Optimal objective  5.500000000e+03\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "pd: {flow: 7, parameters: 1}"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "sln_pd = netflow_pd.solve(dat_pd)\n",
+    "sln_pd"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
    "metadata": {},
    "outputs": [
     {
@@ -304,86 +612,145 @@
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
-       "      <th></th>\n",
-       "      <th></th>\n",
-       "      <th>Quantity</th>\n",
+       "      <th>Parameter</th>\n",
+       "      <th>Value</th>\n",
        "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
        "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>Total Cost</td>\n",
+       "      <td>5500.0</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "    Parameter   Value\n",
+       "0  Total Cost  5500.0"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "sln_pd.parameters"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
        "      <th>Commodity</th>\n",
        "      <th>Source</th>\n",
        "      <th>Destination</th>\n",
-       "      <th></th>\n",
+       "      <th>Quantity</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
-       "      <td rowspan=\"5\" valign=\"top\">Pencils</td>\n",
-       "      <td rowspan=\"3\" valign=\"top\">Denver</td>\n",
-       "      <td>Boston</td>\n",
-       "      <td>51.0</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <td>New York</td>\n",
-       "      <td>149.0</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>Pencils</td>\n",
+       "      <td>Denver</td>\n",
        "      <td>Seattle</td>\n",
-       "      <td>40.0</td>\n",
+       "      <td>10.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <td rowspan=\"2\" valign=\"top\">Detroit</td>\n",
-       "      <td>Boston</td>\n",
-       "      <td>149.0</td>\n",
+       "      <th>1</th>\n",
+       "      <td>Pens</td>\n",
+       "      <td>Denver</td>\n",
+       "      <td>Seattle</td>\n",
+       "      <td>30.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>Pens</td>\n",
+       "      <td>Detroit</td>\n",
        "      <td>New York</td>\n",
-       "      <td>51.0</td>\n",
+       "      <td>30.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <td rowspan=\"4\" valign=\"top\">Pens</td>\n",
-       "      <td rowspan=\"2\" valign=\"top\">Denver</td>\n",
+       "      <th>3</th>\n",
+       "      <td>Pens</td>\n",
+       "      <td>Detroit</td>\n",
        "      <td>Boston</td>\n",
-       "      <td>40.0</td>\n",
+       "      <td>30.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <td>Seattle</td>\n",
-       "      <td>120.0</td>\n",
+       "      <th>4</th>\n",
+       "      <td>Pencils</td>\n",
+       "      <td>Detroit</td>\n",
+       "      <td>Boston</td>\n",
+       "      <td>50.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <td rowspan=\"2\" valign=\"top\">Detroit</td>\n",
+       "      <th>5</th>\n",
+       "      <td>Pens</td>\n",
+       "      <td>Denver</td>\n",
        "      <td>Boston</td>\n",
-       "      <td>120.0</td>\n",
+       "      <td>10.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
+       "      <th>6</th>\n",
+       "      <td>Pencils</td>\n",
+       "      <td>Denver</td>\n",
        "      <td>New York</td>\n",
-       "      <td>120.0</td>\n",
+       "      <td>50.0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "text/plain": [
-       "                               Quantity\n",
-       "Commodity Source  Destination          \n",
-       "Pencils   Denver  Boston           51.0\n",
-       "                  New York        149.0\n",
-       "                  Seattle          40.0\n",
-       "          Detroit Boston          149.0\n",
-       "                  New York         51.0\n",
-       "Pens      Denver  Boston           40.0\n",
-       "                  Seattle         120.0\n",
-       "          Detroit Boston          120.0\n",
-       "                  New York        120.0"
+       "  Commodity   Source Destination  Quantity\n",
+       "0   Pencils   Denver     Seattle      10.0\n",
+       "1      Pens   Denver     Seattle      30.0\n",
+       "2      Pens  Detroit    New York      30.0\n",
+       "3      Pens  Detroit      Boston      30.0\n",
+       "4   Pencils  Detroit      Boston      50.0\n",
+       "5      Pens   Denver      Boston      10.0\n",
+       "6   Pencils   Denver    New York      50.0"
       ]
      },
-     "execution_count": 7,
+     "execution_count": 15,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
    "source": [
-    "sln_pandas = solution_schema.copy_to_pandas(sln)\n",
-    "sln_pandas.flow"
+    "sln_pd.flow"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As you can see, `sln_pd` is the same collection of `DataFrame` results as `sln_to_pandas`, and thus converting into list-of-lists would similarly be trivial."
    ]
   },
   {
@@ -392,14 +759,14 @@
    "source": [
     "## Using ticdat to build robust engines\n",
     "\n",
-    "The preceding section demonstrated how we can use `ticdat` to build modular engines. We now demonstrate how we can use `ticdat` to build engines that check `solve` pre-conditions, and are thus robust with respect to data integrity problems.\n",
+    "We have demonstrated how we can use `ticdat` to build modular engines. We now demonstrate how we can use `ticdat` to build engines that check `solve` pre-conditions, and are thus robust with respect to data integrity problems.\n",
     "\n",
     "First, lets violate our (somewhat artificial) rule that the commodity volume must be positive."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 16,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -415,7 +782,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 17,
    "metadata": {},
    "outputs": [
     {
@@ -424,13 +791,13 @@
        "{TableField(table='commodities', field='Volume'): ValuesPks(bad_values=(0,), pks=('Pens',))}"
       ]
      },
-     "execution_count": 9,
+     "execution_count": 17,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
    "source": [
-    "input_schema.find_data_type_failures(dat)"
+    "netflow.input_schema.find_data_type_failures(dat)"
    ]
   },
   {
@@ -442,7 +809,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 18,
    "metadata": {},
    "outputs": [
     {
@@ -452,14 +819,14 @@
        "  (('Crayons', 'Detroit', 'Seattle'),))}"
       ]
      },
-     "execution_count": 10,
+     "execution_count": 18,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
    "source": [
     "dat.cost['Crayons', 'Detroit', 'Seattle'] = 10\n",
-    "input_schema.find_foreign_key_failures(dat, verbosity=\"Low\")"
+    "netflow.input_schema.find_foreign_key_failures(dat, verbosity=\"Low\")"
    ]
   },
   {
@@ -468,6 +835,186 @@
    "source": [
     "In real life, data integrity failures can typically be grouped into a small number of categories. However, the number of failures in each category might be quite large. `ticdat` creates data structures for each of these categories that can themselves be examined programmatically. As a result, an analyst can leverage the power of Python to detect patterns in the data integrity problems."
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Robust engines with PanDatFactory\n",
+    "As before, we will demonstrate the same sort of integrity checks, except this time using a `solve` engine that is based on `DataFrame` input and thus uses `PanDatFactory`.\n",
+    "\n",
+    "First, lets make a `DataFrame` copy of the `dat` object that has integrity violations."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dat_pd = netflow.input_schema.copy_to_pandas(dat, reset_index=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, lets check the two types of integrity failures. Bear in mind that `find_data_type_failures` and `find_foreign_key_failures` will now return dictionaries that themselves contain `DataFrame` objects, so we will pretty them up a bit for display."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "dict_keys([TableField(table='commodities', field='Volume')])"
+      ]
+     },
+     "execution_count": 20,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "data_type_fais = netflow_pd.input_schema.find_data_type_failures(dat_pd)\n",
+    "data_type_fais.keys()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>Name</th>\n",
+       "      <th>Volume</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>Pens</td>\n",
+       "      <td>0.0</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "   Name  Volume\n",
+       "1  Pens     0.0"
+      ]
+     },
+     "execution_count": 21,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "data_type_fais['commodities', 'Volume']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "dict_keys([('cost', 'commodities', ('Commodity', 'Name'))])"
+      ]
+     },
+     "execution_count": 22,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "fk_fails = netflow_pd.input_schema.find_foreign_key_failures(dat_pd, verbosity=\"Low\")\n",
+    "fk_fails.keys()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>Commodity</th>\n",
+       "      <th>Source</th>\n",
+       "      <th>Destination</th>\n",
+       "      <th>Cost</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>12</th>\n",
+       "      <td>Crayons</td>\n",
+       "      <td>Detroit</td>\n",
+       "      <td>Seattle</td>\n",
+       "      <td>10.0</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "   Commodity   Source Destination  Cost\n",
+       "12   Crayons  Detroit     Seattle  10.0"
+      ]
+     },
+     "execution_count": 23,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "fk_fails['cost', 'commodities', ('Commodity', 'Name')]"
+   ]
   }
  ],
  "metadata": {
diff --git a/examples/gurobipy/netflow/netflow_pd.py b/examples/gurobipy/netflow/netflow_pd.py
new file mode 100644
index 00000000..353b1fe4
--- /dev/null
+++ b/examples/gurobipy/netflow/netflow_pd.py
@@ -0,0 +1,111 @@
+#
+# Solve a multi-commodity flow problem.
+#
+# See netflow.py for more information. This file is similar, except it uses PanDatFactory
+# instead of TicDatFactory. As a result, the solve function takes a PanDat object, which is simply
+# a collection of pre-validated pandas.DataFrame objects consistent with the input_schema.
+#
+# If your default methodology is to convert input data into pandas.DataFrame objects, then PanDatFactory
+# can dramatically help you pre-define the pre and post conditions of your solve subroutine.
+#
+try:
+    import gurobipy as gp
+except:
+    gp = None
+from ticdat import PanDatFactory, standard_main, Slicer
+
+# ------------------------ define the input schema --------------------------------
+input_schema = PanDatFactory (
+    commodities=[["Name"], ["Volume"]],
+    nodes  = [["Name"],[]],
+     arcs = [["Source", "Destination"],["Capacity"]],
+     cost = [["Commodity", "Source", "Destination"], ["Cost"]],
+     inflow = [["Commodity", "Node"],["Quantity"]]
+)
+
+# Define the foreign key relationships
+input_schema.add_foreign_key("arcs", "nodes", ['Source', 'Name'])
+input_schema.add_foreign_key("arcs", "nodes", ['Destination', 'Name'])
+input_schema.add_foreign_key("cost", "arcs", [['Source', 'Source'], ['Destination', 'Destination']])
+input_schema.add_foreign_key("cost", "nodes", ['Destination', 'Name'])
+input_schema.add_foreign_key("cost", "commodities", ['Commodity', 'Name'])
+input_schema.add_foreign_key("inflow", "commodities", ['Commodity', 'Name'])
+input_schema.add_foreign_key("inflow", "nodes", ['Node', 'Name'])
+
+# Define the data types
+input_schema.set_data_type("arcs", "Capacity", min=0, max=float("inf"),
+                           inclusive_min=True, inclusive_max=True)
+input_schema.set_data_type("cost", "Cost", min=0, max=float("inf"),
+                           inclusive_min=True, inclusive_max=False)
+input_schema.set_data_type("inflow", "Quantity", min=-float("inf"), max=float("inf"),
+                           inclusive_min=False, inclusive_max=False)
+input_schema.set_data_type("commodities", "Volume", min=0, max=float("inf"),
+                           inclusive_min=False, inclusive_max=False)
+
+# The default-default of zero makes sense everywhere except for Capacity
+input_schema.set_default_value("arcs", "Capacity", float("inf"))
+# ---------------------------------------------------------------------------------
+
+# ------------------------ define the output schema -------------------------------
+solution_schema = PanDatFactory(
+        flow = [["Commodity", "Source", "Destination"], ["Quantity"]],
+        parameters = [["Parameter"],["Value"]])
+# ---------------------------------------------------------------------------------
+
+# ------------------------ solving section-----------------------------------------
+def solve(dat):
+    """
+    core solving routine
+    :param dat: a good ticdat for the dataFactory
+    :return: a good ticdat for the solutionFactory, or None
+    """
+    assert input_schema.good_pan_dat_object(dat)
+    # PanDat objects might contain duplicate rows, whereas TicDat objects cannot. This is because DataFrames
+    # can contain duplicate rows, but dictionaries cannot.
+    assert not input_schema.find_duplicates(dat)
+    assert not input_schema.find_foreign_key_failures(dat)
+    assert not input_schema.find_data_type_failures(dat)
+
+
+    # Create optimization model
+    mdl = gp.Model('netflow')
+
+    # itertuples is the most performant way to iterate over the rows of a DataFrame
+    flow = {(h, i, j): mdl.addVar(name=f'flow_{h}_{i}_{j}', obj=cost)
+            for h, i, j, cost in dat.cost.itertuples(index=False)}
+
+    flowslice = Slicer(flow)
+    volume = {k: volume for k, volume in dat.commodities.itertuples(index=False)}
+
+    # Arc Capacity constraints
+    for i, j, capacity in dat.arcs.itertuples(index=False):
+        mdl.addConstr(gp.quicksum(flow[_h, _i, _j] * volume[_h]
+                                  for _h, _i, _j in flowslice.slice('*', i, j))
+                      <= capacity, name=f'cap_{i}_{j}')
+
+    inflow = {(h, j): qty for h, j, qty in dat.inflow.itertuples(index=False) if abs(qty) > 0}
+    # Flow conservation constraints. Constraints are generated only for relevant pairs.
+    # So we generate a conservation of flow constraint if there is negative or positive inflow
+    # quantity, or at least one inbound flow variable, or at least one outbound flow variable.
+    for h, j in set(inflow).union({(h, i) for h, i, j in flow}, {(h, j) for h, i, j in flow}):
+        mdl.addConstr(
+            gp.quicksum(flow[h_i_j] for h_i_j in flowslice.slice(h, '*', j)) +
+            inflow.get((h, j), 0) ==
+            gp.quicksum(flow[h_j_i] for h_j_i in flowslice.slice(h, j, '*')),
+            name=f'node_{h}_{j}')
+
+    # Compute optimal solution
+    mdl.optimize()
+
+    if mdl.status == gp.GRB.status.OPTIMAL:
+        # PanDatFactory also makes it easy to create DataFrame objects from rows of data
+        rtn = solution_schema.PanDat(flow=[[h, i, j, var.x] for (h, i, j), var in flow.items() if var.x > 0],
+                                     parameters=[["Total Cost",  mdl.objVal]])
+        return rtn
+# ---------------------------------------------------------------------------------
+
+# ------------------------ provide stand-alone functionality ----------------------
+# when run from the command line, will read/write xls/csv/db/mdb files
+if __name__ == "__main__":
+    standard_main(input_schema, solution_schema, solve)
+# ---------------------------------------------------------------------------------
diff --git a/examples/gurobipy/readme.md b/examples/gurobipy/readme.md
index 5e428b3b..97865ffa 100644
--- a/examples/gurobipy/readme.md
+++ b/examples/gurobipy/readme.md
@@ -1,7 +1,9 @@
-Repository for gurobipy ticdat examples.
+Repository for `gurobipy` `ticdat` examples. The bulk of these examples use `TicDatFactory`, 
+and thus render input tables in dict-of-dicts format. `netflow_pd` demonstrates an engine that accepts
+input tables as `pandas.DataFrame` objects, and thus uses `PanDatFactory`.
 
  * diet - standard diet example.
- * netflow - standard multicommodity network flow problems
+ * netflow - standard multicommodity network flow.
  * fantop - fantasy football example. See [here](https://www.linkedin.com/pulse/fantasy-footballers-nerds-too-peter-cacioppi/) for more information.
  * metrorail - optimize the number of times you visit the fare kiosk when riding the metro. See [here](https://www.linkedin.com/pulse/miami-metrorail-meets-python-peter-cacioppi/) for more information. This is a good example of constructing a comprehensive solution from a series of sub-problems.
  * workforce - Demonstrates lexicographical optimization for a workforce scheduling problem.
diff --git a/examples/gurobipy/simplest_examples/diet.py b/examples/gurobipy/simplest_examples/diet.py
index 51a83b8a..24324e2a 100644
--- a/examples/gurobipy/simplest_examples/diet.py
+++ b/examples/gurobipy/simplest_examples/diet.py
@@ -1,9 +1,9 @@
 # Simplest diet example using gurobipy and ticdat
 
 try: # if you don't have gurobipy installed, the code will still load and then fail on solve
-    import gurobipy as gu
+    import gurobipy as gp
 except:
-    gu = None
+    gp = None
 from ticdat import TicDatFactory, standard_main
 
 input_schema = TicDatFactory (
@@ -19,7 +19,7 @@
 def solve(dat):
     assert input_schema.good_tic_dat_object(dat)
 
-    mdl = gu.Model("diet")
+    mdl = gp.Model("diet")
 
     nutrition = {c:mdl.addVar(lb=n["Min Nutrition"], ub=n["Max Nutrition"], name=c)
                 for c,n in dat.categories.items()}
@@ -29,15 +29,15 @@ def solve(dat):
 
      # Nutrition constraints
     for c in dat.categories:
-        mdl.addConstr(gu.quicksum(dat.nutrition_quantities[f,c]["Quantity"] * buy[f]
-                      for f in dat.foods) == nutrition[c],
+        mdl.addConstr(gp.quicksum(dat.nutrition_quantities[f, c]["Quantity"] * buy[f]
+                                  for f in dat.foods) == nutrition[c],
                       name = c)
 
-    mdl.setObjective(gu.quicksum(buy[f] * c["Cost"] for f,c in dat.foods.items()),
-                     sense=gu.GRB.MINIMIZE)
+    mdl.setObjective(gp.quicksum(buy[f] * c["Cost"] for f, c in dat.foods.items()),
+                     sense=gp.GRB.MINIMIZE)
     mdl.optimize()
 
-    if mdl.status == gu.GRB.OPTIMAL:
+    if mdl.status == gp.GRB.OPTIMAL:
         sln = solution_schema.TicDat()
         for f,x in buy.items():
             if x.x > 0:
diff --git a/examples/gurobipy/simplest_examples/netflow.py b/examples/gurobipy/simplest_examples/netflow.py
index 854c0e4c..93f3e167 100644
--- a/examples/gurobipy/simplest_examples/netflow.py
+++ b/examples/gurobipy/simplest_examples/netflow.py
@@ -1,9 +1,9 @@
 # Simplest netflow example using gurobipy and ticdat
 
 try: # if you don't have gurobipy installed, the code will still load and then fail on solve
-    import gurobipy as gu
+    import gurobipy as gp
 except:
-    gu = None
+    gp = None
 from ticdat import TicDatFactory, standard_main, Slicer
 
 input_schema = TicDatFactory (
@@ -21,7 +21,7 @@
 def solve(dat):
     assert input_schema.good_tic_dat_object(dat)
 
-    mdl = gu.Model("netflow")
+    mdl = gp.Model("netflow")
 
     flow = {(h, i, j): mdl.addVar(name='flow_%s_%s_%s' % (h, i, j)) for h, i, j in dat.cost}
 
@@ -29,7 +29,7 @@ def solve(dat):
 
     # Arc Capacity constraints
     for i,j in dat.arcs:
-        mdl.addConstr(gu.quicksum(flow[_h, _i, _j] * dat.commodities[_h]["Volume"]
+        mdl.addConstr(gp.quicksum(flow[_h, _i, _j] * dat.commodities[_h]["Volume"]
                                   for _h, _i, _j in flowslice.slice('*', i, j))
                       <= dat.arcs[i,j]["Capacity"],
                       name='cap_%s_%s' % (i, j))
@@ -40,17 +40,17 @@ def solve(dat):
     for h in dat.commodities:
         for j in dat.nodes:
             mdl.addConstr(
-                gu.quicksum(flow[h_i_j] for h_i_j in flowslice.slice(h,'*',j)) +
+                gp.quicksum(flow[h_i_j] for h_i_j in flowslice.slice(h, '*', j)) +
                 dat.inflow.get((h,j), {"Quantity":0})["Quantity"] ==
-                gu.quicksum(flow[h_j_i] for h_j_i in flowslice.slice(h, j, '*')),
+                gp.quicksum(flow[h_j_i] for h_j_i in flowslice.slice(h, j, '*')),
                 name='node_%s_%s' % (h, j))
 
-    mdl.setObjective(gu.quicksum(flow * dat.cost[h, i, j]["Cost"]
+    mdl.setObjective(gp.quicksum(flow * dat.cost[h, i, j]["Cost"]
                                  for (h, i, j), flow in flow.items()),
-                     sense=gu.GRB.MINIMIZE)
+                     sense=gp.GRB.MINIMIZE)
     mdl.optimize()
 
-    if mdl.status == gu.GRB.OPTIMAL:
+    if mdl.status == gp.GRB.OPTIMAL:
         rtn = solution_schema.TicDat()
         for (h, i, j), var in flow.items():
             if var.x > 0:
diff --git a/examples/gurobipy/workforce/workforce.py b/examples/gurobipy/workforce/workforce.py
index 19a837c3..117f56a0 100644
--- a/examples/gurobipy/workforce/workforce.py
+++ b/examples/gurobipy/workforce/workforce.py
@@ -7,7 +7,7 @@
 # tries to balance the workload among the workers.
 #
 
-import gurobipy as gu
+import gurobipy as gp
 from ticdat import TicDatFactory,  standard_main, gurobi_env
 
 # ------------------------ define the input schema --------------------------------
@@ -47,10 +47,10 @@ def solve(dat):
     assert not input_schema.find_foreign_key_failures(dat)
     assert not input_schema.find_data_type_failures(dat)
 
-    m = gu.Model("assignment", env=gurobi_env())
+    m = gp.Model("assignment", env=gurobi_env())
 
     # Assignment variables: x[w,s] == 1 if worker w is assigned to shift s.
-    x = m.addVars(dat.availability, vtype=gu.GRB.BINARY, name="x")
+    x = m.addVars(dat.availability, vtype=gp.GRB.BINARY, name="x")
 
     # Slack variables for each shift constraint
     slacks = m.addVars(dat.shifts, name="Slack")
@@ -72,9 +72,9 @@ def solve(dat):
     m.setObjective(totSlack)
     def _solve():
         m.optimize()
-        if m.status in [gu.GRB.Status.INF_OR_UNBD, gu.GRB.Status.INFEASIBLE, gu.GRB.Status.UNBOUNDED]:
+        if m.status in [gp.GRB.Status.INF_OR_UNBD, gp.GRB.Status.INFEASIBLE, gp.GRB.Status.UNBOUNDED]:
             print('The model cannot be solved because it is infeasible or unbounded')
-        elif m.status != gu.GRB.Status.OPTIMAL:
+        elif m.status != gp.GRB.Status.OPTIMAL:
             print('Optimization was stopped with status %d' % m.status)
         else:
             return True
@@ -83,7 +83,7 @@ def _solve():
         totSlack.ub = totSlack.lb = totSlack.x
 
         totalPayments = m.addVar(name="totalPayments")
-        m.addConstr(totalPayments == gu.quicksum(dat.workers[w]["Payment"]*x[w,s]
+        m.addConstr(totalPayments == gp.quicksum(dat.workers[w]["Payment"] * x[w, s]
                                                  for w,s in dat.availability))
         m.setObjective(totalPayments)
         if _solve():
@@ -94,7 +94,7 @@ def _solve():
 
             # Variables to count the difference from average for each worker;
             # note that these variables can take negative values.
-            diffShifts = m.addVars(dat.workers, lb=-gu.GRB.INFINITY, name="Diff")
+            diffShifts = m.addVars(dat.workers, lb=-gp.GRB.INFINITY, name="Diff")
 
             # Constraint: compute the average number of shifts worked
             m.addConstr(len(dat.workers) * avgShifts == totShifts.sum(), "avgShifts")
@@ -105,7 +105,7 @@ def _solve():
 
             # Objective: minimize the sum of the square of the difference from the
             # average number of shifts worked
-            m.setObjective(gu.quicksum(diffShifts[w]*diffShifts[w] for w in dat.workers))
+            m.setObjective(gp.quicksum(diffShifts[w] * diffShifts[w] for w in dat.workers))
             if _solve():
                 sln = solution_schema.TicDat()
                 for (w,s),x_var in x.items():
diff --git a/setup.py b/setup.py
index a18c9bee..69953420 100644
--- a/setup.py
+++ b/setup.py
@@ -12,7 +12,7 @@
 	long_description_content_type='text/markdown',
 	license = 'BSD 2-Clause',
 	author = 'Pete Cacioppi',
-	author_email= 'peter.cacioppi@opexanalytics.com',
+	author_email= 'peter.cacioppi@gmail.com',
 	maintainer_email = '12samn@gmail.com',
 	url = 'https://github.com/ticdat/ticdat/',
 	classifiers = [
diff --git a/ticdat/__init__.py b/ticdat/__init__.py
index 200002c6..2cd414d3 100644
--- a/ticdat/__init__.py
+++ b/ticdat/__init__.py
@@ -10,11 +10,12 @@
 
 from ticdat.ticdatfactory import TicDatFactory, freeze_me
 from ticdat.utils import Sloc, LogFile, Progress, Slicer, standard_main, \
-                         gurobi_env, ampl_format, verify, dat_restricted, sln_restricted
+                         gurobi_env, ampl_format, verify, dat_restricted, sln_restricted, \
+                         faster_df_apply
 from ticdat.opl import opl_run, create_opl_mod_text, create_opl_mod_output_text
 from ticdat.model import Model
 from ticdat.pandatfactory import PanDatFactory
-__version__ = '0.2.21'
+__version__ = '0.2.21.1'
 __all__ = ["TicDatFactory", "PanDatFactory", "freeze_me", "LogFile", "Sloc", "Slicer", "Progress", "standard_main",
            "Model", "opl_run", "create_opl_mod_text", "create_opl_mod_output_text", "ampl_format",
-           "gurobi_env", "verify", "dat_restricted", "sln_restricted"]
+           "gurobi_env", "verify", "dat_restricted", "sln_restricted", "faster_df_apply"]
diff --git a/ticdat/jsontd.py b/ticdat/jsontd.py
index 39599f31..61640788 100644
--- a/ticdat/jsontd.py
+++ b/ticdat/jsontd.py
@@ -7,6 +7,7 @@
 from ticdat.utils import freezable_factory, TicDatError, verify, stringish, dictish, containerish
 from ticdat.utils import find_duplicates_from_dict_ticdat
 import datetime
+import itertools
 
 try:
     import json
@@ -60,6 +61,9 @@ def __init__(self, tic_dat_factory):
         """
         self.tic_dat_factory = tic_dat_factory
         self._isFrozen = True
+    def _looks_pandas(self, jdict):
+        if not all(set(itertools.chain(*v)) == {'columns', 'data'} for v in self.tic_dat_factory.schema().values()):
+            return all(dictish(v) and set(v.keys()) == {'columns', 'data'} for v in  jdict.values())
     def create_tic_dat(self, json_file_path, freeze_it = False, from_pandas = False):
         """
         Create a TicDat object from a json file
@@ -70,7 +74,9 @@ def create_tic_dat(self, json_file_path, freeze_it = False, from_pandas = False)
         :param freeze_it: boolean. should the returned object be frozen?
 
         :param from_pandas: boolean.  If truthy, then use pandas json readers. See
-                            PanDatFactory json readers for more details.
+                            PanDatFactory json readers for more details. This argument is historical, as a
+                            json format that matches the PanDatFactory.json format will be detected automatically,
+                            and thus client code is generally safe ignoring this argument completely.
 
         :return: a TicDat object populated by the matching tables.
 
@@ -84,8 +90,10 @@ def create_tic_dat(self, json_file_path, freeze_it = False, from_pandas = False)
             from ticdat import PanDatFactory
             pdf = PanDatFactory.create_from_full_schema(self.tic_dat_factory.schema(include_ancillary_info=True))
             _rtn = pdf.json.create_pan_dat(json_file_path)
-            return pdf.copy_to_tic_dat(_rtn)
+            return pdf.copy_to_tic_dat(_rtn, freeze_it=freeze_it)
         jdict = self._create_jdict(json_file_path)
+        if self._looks_pandas(jdict):
+            return self.create_tic_dat(json_file_path, freeze_it=freeze_it, from_pandas=True)
         tic_dat_dict = self._create_tic_dat_dict(jdict)
         missing_tables = set(self.tic_dat_factory.all_tables).difference(tic_dat_dict)
         if missing_tables:
@@ -120,6 +128,8 @@ def find_duplicates(self, json_file_path, from_pandas = False):
             jdict = {t: [tuple(_) for _ in getattr(_rtn, t).itertuples(index=False)] for t in pdf.all_tables}
         else:
             jdict = self._create_jdict(json_file_path)
+            if self._looks_pandas(jdict):
+                return self.find_duplicates(json_file_path, from_pandas=True)
         rtn = find_duplicates_from_dict_ticdat(self.tic_dat_factory, jdict)
         return rtn or {}
     def _create_jdict(self, path_or_buf):
@@ -168,7 +178,7 @@ def _create_tic_dat_dict(self, jdict):
                 else:
                     rtn[t].append([tdf._general_read_cell(t, f, x) for f, x in zip(all_fields, row)])
         return rtn
-    def write_file(self, tic_dat, json_file_path, allow_overwrite = False, verbose = False, to_pandas = False):
+    def write_file(self, tic_dat, json_file_path, allow_overwrite=False, verbose=False, to_pandas=False):
         """
         write the ticDat data to a json file (or json string)
 
diff --git a/ticdat/pandatfactory.py b/ticdat/pandatfactory.py
index 2be20c5c..e1724bc5 100644
--- a/ticdat/pandatfactory.py
+++ b/ticdat/pandatfactory.py
@@ -23,27 +23,12 @@
 
 pd, DataFrame = utils.pd, utils.DataFrame # if pandas not installed will be falsey
 
-def _faster_df_apply(df, func, trip_wire_check=None):
-    cols = list(df.columns)
-    data, index = [], []
-    for row in df.itertuples(index=True):
-        row_dict = {f:v for f,v in zip(cols, row[1:])}
-        data.append(func(row_dict))
-        index.append(row[0])
-        if trip_wire_check:
-            new_func = trip_wire_check(data[-1])
-            if new_func:
-                func = new_func
-                trip_wire_check = None
-    # will default to float for empty Series, like original pandas
-    return pd.Series(data, index=index, **({"dtype": numpy.float64} if not data else {}))
-
 def _is_last_rows_all_nan(df, last_rows):
     assert last_rows > 0
     # quick last row check to make faster
     all_nan_row =  lambda row: all(map(pd.isnull, row.values()))
-    return _faster_df_apply(df.tail(1), all_nan_row).all() and \
-           _faster_df_apply(df.tail(last_rows), all_nan_row,
+    return utils.faster_df_apply(df.tail(1), all_nan_row).all() and \
+           utils.faster_df_apply(df.tail(last_rows), all_nan_row,
                             trip_wire_check=lambda x: all_nan_row if x else lambda row: False).all()
 
 def _find_number_all_nan_last_rows(df, min_rtn, max_rtn):
@@ -177,14 +162,21 @@ def clone(self, table_restrictions=None, clone_factory=None):
         """
 
         clones the PanDatFactory
+
         :param table_restrictions : if None, then argument is ignored. Otherwise, a container listing the
                                     tables to keep in the clone. Tables outside table_restrictions are removed from
                                     the clone.
+
         :param clone_factory : optional. Defaults to PanDatFactory. Can also be TicDatFactory.  Can also be a function,
                                in which case it should behave similarly to create_from_full_schema.
+                               If clone_factory=TicDatFactory, the row predicates that use predicate_kwargs_maker
+                               won't be copied over.
+
         :return: a clone of the PanDatFactory. Returned object will be based on clone_factory, if provided.
         """
         clone_factory = clone_factory or PanDatFactory
+        from ticdat import TicDatFactory
+        no_copy_predicate_kwargs_maker = clone_factory == TicDatFactory
         if hasattr(clone_factory, "create_from_full_schema"):
             clone_factory = clone_factory.create_from_full_schema
         full_schema = utils.clone_a_anchillary_info_schema(self.schema(include_ancillary_info=True), table_restrictions)
@@ -192,9 +184,10 @@ def clone(self, table_restrictions=None, clone_factory=None):
         for tbl, row_predicates in self._data_row_predicates.items():
             if table_restrictions is None or tbl in table_restrictions:
                 for pn, rpi in row_predicates.items():
-                    rtn.add_data_row_predicate(tbl, predicate=rpi.predicate, predicate_name=pn,
-                                               predicate_kwargs_maker=rpi.predicate_kwargs_maker,
-                                               predicate_failure_response=rpi.predicate_failure_response)
+                    if not (rpi.predicate_kwargs_maker and no_copy_predicate_kwargs_maker):
+                        rtn.add_data_row_predicate(tbl, predicate=rpi.predicate, predicate_name=pn,
+                                                   predicate_kwargs_maker=rpi.predicate_kwargs_maker,
+                                                   predicate_failure_response=rpi.predicate_failure_response)
         return rtn
     @property
     def default_values(self):
@@ -300,7 +293,7 @@ def _general_post_read_adjustment(self, dat, push_parameters_to_be_valid=False,
         :param json_read: special 'None'->None override needed for pandas json reader
         '''
         assert push_parameters_to_be_valid or not json_read, "json_read should always push_parameters_to_be_valid"
-        apply = _faster_df_apply
+        apply = utils.faster_df_apply
         for t in set(self.all_tables).difference(["parameters"]): # parameters table is handled differently
             df = getattr(dat, t)
             for f in self.primary_key_fields.get(t, ()) + self.data_fields.get(t, ()):
@@ -353,7 +346,7 @@ def fix_value(row):
                 if number_v is not None and safe_apply(int)(number_v) == number_v:
                     number_v = int(number_v)
                 return value if number_v is None else number_v
-            dat.parameters[val_fld] = _faster_df_apply(dat.parameters, lambda row: fix_value(row))
+            dat.parameters[val_fld] = utils.faster_df_apply(dat.parameters, lambda row: fix_value(row))
         return dat
     def _pre_write_adjustment(self, dat):
         '''
@@ -372,11 +365,11 @@ def _pre_write_adjustment(self, dat):
         rtn = self.copy_pan_dat(rtn) # deep copy so data changes don't side effect
         if self.parameters: # Assuming a parameters table without parameters specification is just a naive developer
             fld = self.data_fields["parameters"][0]
-            rtn.parameters[fld] = _faster_df_apply(rtn.parameters,
-                                                   lambda row: None if isnull(row[fld]) else str(row[fld]))
+            rtn.parameters[fld] = utils.faster_df_apply(rtn.parameters,
+                                                   lambda row: None if isnull(row[fld]) else row[fld])
         if self.infinity_io_flag == "N/A":
             return rtn
-        apply = _faster_df_apply
+        apply = utils.faster_df_apply
         for t in set(self.all_tables).difference(["parameters"]): # parameters table is handled differently
             df = getattr(rtn, t)
             for f in self.primary_key_fields.get(t, ()) + self.data_fields.get(t, ()):
@@ -794,10 +787,18 @@ def __init__(self, **init_tables):
                            "Failed to provide a valid DataFrame or DataFrame construction argument for %s"%t)
                     setattr(self, t, tbl.copy())
                     df = getattr(self, t)
+                    pks = superself.primary_key_fields.get(t, ())
+                    if pks and not set(pks).intersection(df.columns) and \
+                       set(pks) == utils.safe_apply(lambda: set(df.index.names))():
+                        df.reset_index(drop=False, inplace=True)
                     if list(df.columns) == list(range(len(df.columns))) and \
                        len(df.columns) >= len(superself._all_fields(t)):
                         df.rename(columns={f1:f2 for f1, f2 in zip(df.columns, superself._all_fields(t))},
                                   inplace=True)
+                    if list(df.columns) != list(range(len(df.columns))):
+                        for f, d in superself.default_values.get(t, {}).items():
+                            if f not in df.columns:
+                                df[f] = d
                 for t in set(superself.all_tables).difference(init_tables):
                     setattr(self, t, DataFrame({f:[] for f in utils.all_fields(superself, t)}))
                 missing_fields = {(t, f) for t in superself.all_tables for f in superself._all_fields(t)
@@ -877,8 +878,6 @@ def copy_to_tic_dat(self, pan_dat, freeze_it=False):
         msg = []
         verify(self.good_pan_dat_object(pan_dat, msg.append),
                "pan_dat not a good object for this factory : %s"%"\n".join(msg))
-        if self.find_duplicates(pan_dat):
-            print("--> Warning: duplicate rows will force copy_to_tic_dat to return an object with fewer total rows")
         rtn = self._copy_to_tic_dat(pan_dat)
         from ticdat import TicDatFactory
         # note - this has a minor problem in that it might downcast pandas.Timestamp to numpy.datetime64
@@ -894,8 +893,6 @@ def _copy_to_tic_dat(self, pan_dat, keep_generics_as_df=True):
         tdf = TicDatFactory(**sch)
         def df(t):
             rtn = getattr(pan_dat, t)
-            if self.primary_key_fields.get(t, ()):
-                return rtn.set_index(list(self.primary_key_fields[t]), drop=False)
             if t in self.generic_tables and not keep_generics_as_df:
                 return list(map(list, rtn.itertuples(index=False)))
             return rtn
@@ -973,7 +970,7 @@ def check_too_many(is_bad_row):
                 def bad_row(row):
                     data = row[field]
                     return not data_type.valid_data(None if isnull(data) else data)
-                where_bad_rows = _faster_df_apply(_table, bad_row, trip_wire_check=check_too_many)
+                where_bad_rows = utils.faster_df_apply(_table, bad_row, trip_wire_check=check_too_many)
                 if where_bad_rows.any():
                     rtn[TableField(table, field)] = _table[where_bad_rows].copy() if as_table else where_bad_rows
                 if number_failures[0] >= max_failures:
@@ -1049,7 +1046,7 @@ def find_data_row_failures(self, pan_dat, as_table=True, exception_handling="__d
               "Unhandled": Exceptions resulting from calling a row predicate (or a predicate_kwargs_maker) will not be
                            handled by data_row_failures.
               "__debug__": Since "Handled as Failure" makes more sense for production runs and "Unhandled" makes more
-                           sense for debugging, this value will use the latter if __debug__ is True and the former
+                           sense for debugging, this option will use the latter if __debug__ is True and the former
                            otherwise. See -o and __debug__ in Python documentation for more details.
 
         :param max_failures: number. An upper limit on the number of failures to find. Will short circuit and return
@@ -1075,15 +1072,15 @@ def find_data_row_failures(self, pan_dat, as_table=True, exception_handling="__d
         number_failures = [0]
         check_too_many_bool = check_too_many_msg = None
         if max_failures < float("inf"):
-            def check_too_many_bool(is_bad_row):  # here _faster_df_apply is applying a function that
+            def check_too_many_bool(is_bad_row):  # here faster_df_apply is applying a function that
                 if is_bad_row:  # returns True when the row is bad and False o/wise (the boolean reverse of the
                     number_failures[0] += 1 # of the row predicate)
                     if number_failures[0] >= max_failures:  # all future rows will be good
                         return lambda row: False  # which in this context is not bad (i.e. False)
 
-            def check_too_many_msg(true_or_msg):  # here _faster_df_apply is applying a function that
+            def check_too_many_msg(true_or_msg):  # here faster_df_apply is applying a function that
                 if true_or_msg is not True:  # returns True when the row is good and a string o/wise
-                    number_failures[0] += 1  # (i.e. _faster_df_applyu is using the actual row predicate)
+                    number_failures[0] += 1  # (i.e. faster_df_applyu is using the actual row predicate)
                     if number_failures[0] >= max_failures:  # all future rows will be good
                         return lambda row: True  # which in this context is True
 
@@ -1143,7 +1140,7 @@ def _p(row):
                         bad_row = (lambda row: not rpi.predicate(row, **predicate_kwargs)) \
                                   if exception_handling == "Unhandled" else (lambda row: not _p(row))
 
-                        where_bad_rows = _faster_df_apply(_table, bad_row, trip_wire_check=check_too_many_bool)
+                        where_bad_rows = utils.faster_df_apply(_table, bad_row, trip_wire_check=check_too_many_bool)
                         if where_bad_rows.any():
                             rtn[TPN(tbl, pn)] = _table[where_bad_rows].copy() if as_table else where_bad_rows
                     else:
@@ -1154,7 +1151,7 @@ def _p(row):
                                 return f"Exception<{e}>"
                         predicate = (lambda row: rpi.predicate(row, **predicate_kwargs)) \
                                     if exception_handling == "Unhandled" else (lambda row: _p(row))
-                        predicate_result = _faster_df_apply(_table, predicate, trip_wire_check=check_too_many_msg)
+                        predicate_result = utils.faster_df_apply(_table, predicate, trip_wire_check=check_too_many_msg)
                         where_bad_rows = predicate_result.apply(lambda x: x is not True)
                         if where_bad_rows.any():
                             if as_table:
@@ -1254,8 +1251,8 @@ def check_too_many(is_bad_row):
                 # I'm casting to list here because child is a copy that doesn't have the original index
                 # This is fixable, (i.e. we could return a Series with an index matching the original child table)
                 # but the fix isn't high priority.
-                rtn[fk] = list(_faster_df_apply(child, lambda row: row[magic_field*2] in bad_rows,
-                                                trip_wire_check=check_too_many))
+                rtn[fk] = list(utils.faster_df_apply(child, lambda row: row[magic_field*2] in bad_rows,
+                                                     trip_wire_check=check_too_many))
                 if number_failures[0] >= max_failures:
                     return rtn
         return rtn
diff --git a/ticdat/pandatio.py b/ticdat/pandatio.py
index 6bba2e3f..1fd5d860 100644
--- a/ticdat/pandatio.py
+++ b/ticdat/pandatio.py
@@ -4,12 +4,13 @@
 except:
     sql = None
 import json
-
+import uuid
 import os
 from ticdat.utils import freezable_factory, verify, case_space_to_pretty, pd, TicDatError, FrozenDict, all_fields
-from ticdat.utils import all_underscore_replacements, stringish, dictish, debug_break
+from ticdat.utils import all_underscore_replacements, stringish, dictish, containerish, debug_break, faster_df_apply
 from itertools import product, chain
 from collections import defaultdict
+import datetime
 import inspect
 try:
     import numpy
@@ -31,8 +32,7 @@ def __enter__(self, *execinfo) :
     def __exit__(self, *excinfo) :
         pass
 
-def _clean_pandat_creator(pdf, df_dict, push_parameters_to_be_valid=True, json_read=False,
-                          print_missing_tables=False):
+def _clean_pandat_creator(pdf, df_dict, push_parameters_to_be_valid=True, json_read=False):
     # note that pandas built in IO routines tend to be a bit overy pushy with the typing, hence
     # the push_parameters_to_be_valid argument
     pandat = pdf.PanDat(**df_dict)
@@ -40,8 +40,6 @@ def _clean_pandat_creator(pdf, df_dict, push_parameters_to_be_valid=True, json_r
         flds = [f for f in chain(pdf.primary_key_fields[t], pdf.data_fields[t])]
         setattr(pandat, t, getattr(pandat, t)[flds])
     missing_tables = '\n'.join(sorted({t for t in pdf.all_tables if not len(getattr(pandat, t))}))
-    if missing_tables and print_missing_tables:
-        print(f"The following table names could not be found (or were empty).\n{missing_tables}\n")
     msg = []
     assert pdf.good_pan_dat_object(pandat, msg.append), str(msg)
     return pdf._general_post_read_adjustment(pandat, json_read=json_read,
@@ -75,7 +73,7 @@ def create_pan_dat(self, path_or_buf, fill_missing_fields=False, orient='split',
 
         :param fill_missing_fields: boolean. If truthy, missing fields will be filled in
                                     with their default value. Otherwise, missing fields
-                                    throw an Exception.
+                                    throw an Exception. Doesn't work with list-of-lists format.
 
         :param orient: Indication of expected JSON string format. See pandas.read_json for more details.
 
@@ -103,24 +101,32 @@ def create_pan_dat(self, path_or_buf, fill_missing_fields=False, orient='split',
             verify(stringish(path_or_buf), "%s isn't a string" % path_or_buf)
             loaded_dict = json.loads(path_or_buf)
         verify(dictish(loaded_dict), "the json.load result doesn't resolve to a dictionary")
-        verify(all(map(dictish, loaded_dict.values())),
-               "the json.load result doesn't resolve to a dictionary whose values are themselves dictionaries")
-
         tbl_names = self._get_table_names(loaded_dict)
-        verify("orient" not in kwargs, "orient should be passed as a non-kwargs argument")
-        rtn = {}
-        for t, f in tbl_names.items():
-            kwargs_ = dict(kwargs)
-            if "dtype" not in kwargs_:
-                kwargs_["dtype"] = self.pan_dat_factory._dtypes_for_pandas_read(t)
-            rtn[t] = pd.read_json(json.dumps(loaded_dict[f]), orient=orient, **kwargs_)
-        missing_fields = {(t, f) for t in rtn for f in all_fields(self.pan_dat_factory, t)
-                          if f not in rtn[t].columns}
-        if fill_missing_fields:
-            for t,f in missing_fields:
-                rtn[t][f] = self.pan_dat_factory.default_values[t][f]
-        verify(fill_missing_fields or not missing_fields,
-               "The following (table, field) pairs are missing fields.\n%s" % [(t, f) for t,f in missing_fields])
+        if all(map(containerish, loaded_dict.values())) and not any(map(dictish, loaded_dict.values())):
+            if fill_missing_fields:
+                # there is no obvious assumption for which columns are being supplied and which are missing
+                print("fill_missing_fields isn't appropriate for list-of-lists format.")
+            rtn ={k: loaded_dict[v] for k,v in tbl_names.items()}
+        else:
+            verify(all(map(dictish, loaded_dict.values())),
+                   "the json.load result doesn't resolve to a dictionary whose values are themselves dictionaries")
+
+            verify("orient" not in kwargs, "orient should be passed as a non-kwargs argument")
+            rtn = {}
+            for t, f in tbl_names.items():
+                kwargs_ = dict(kwargs)
+                if "dtype" not in kwargs_:
+                    kwargs_["dtype"] = self.pan_dat_factory._dtypes_for_pandas_read(t)
+                rtn[t] = pd.read_json(json.dumps(loaded_dict[f]), orient=orient, **kwargs_)
+            missing_fields = {(t, f) for t in rtn for f in all_fields(self.pan_dat_factory, t)
+                              if f not in rtn[t].columns}
+            if fill_missing_fields:
+                for t, f in list(missing_fields):
+                    if f in self.pan_dat_factory.default_values.get(t, {}):
+                        rtn[t][f] = self.pan_dat_factory.default_values[t][f]
+                        missing_fields.remove((t, f))
+            verify(not missing_fields,
+                   "The following (table, field) pairs are missing fields.\n%s" % [(t, f) for t,f in missing_fields])
         missing_tables = sorted(set(self.pan_dat_factory.all_tables).difference(rtn))
         if missing_tables:
             print("The following table names could not be found in the json file/string.\n%s\n" %
@@ -137,10 +143,47 @@ def _get_table_names(self, loaded_dict):
             else:
                 rtn.pop(table)
         return rtn
-    def write_file(self, pan_dat, json_file_path, case_space_table_names=False, orient='split',
+    def write_file(self, pan_dat, json_file_path):
+        """
+        Write the PanDat data to a json file (or json string). Writes each table as a list-of-lists.
+        See write_file_pd for other formats.
+
+        :param pan_dat: the PanDat object to write
+
+        :param json_file_path: the json file into which the data is to be written. If falsey, will return a
+                               JSON string
+
+        :return: A JSON string if json_file_path is falsey, otherwise None
+        """
+        msg = []
+        verify(self.pan_dat_factory.good_pan_dat_object(pan_dat, msg.append),
+               "pan_dat not a good object for this factory : %s"%"\n".join(msg))
+        pan_dat = self.pan_dat_factory._pre_write_adjustment(pan_dat)
+        jdict = {}
+        def fix_cell(x):
+            if isinstance(x, (pd.Timestamp, numpy.datetime64, datetime.datetime)):
+                return str(x)
+            if pd.isnull(x):
+                return None
+            return x
+
+        for t, (pks, dfs) in self.pan_dat_factory.schema().items():
+            jdict[t] = []
+            def append_row_list(row):
+                jdict[t].append([fix_cell(row[f]) for f in pks + dfs])
+            faster_df_apply(getattr(pan_dat, t), append_row_list)
+        if not json_file_path:
+            return json.dumps(jdict, sort_keys=True, indent=2)
+        with open(json_file_path, "w") as fp:
+            json.dump(jdict, fp, sort_keys=True, indent=2)
+
+    def write_file_pd(self, pan_dat, json_file_path, case_space_table_names=False, orient='split',
                    index=False, indent=2, sort_keys=False, **kwargs):
         """
-        write the PanDat data to a json file (or json string)
+        write the PanDat data to a json file (or json string).
+        Use this routine to write json text that is consistent with what pandas.to_json.
+        The list-of-lists format is created with write_file. In older ticdat releases, write_file
+        implemented the functionaltiy now provided with write_file_pd.
 
         :param pan_dat: the PanDat object to write
 
@@ -150,7 +193,8 @@ def write_file(self, pan_dat, json_file_path, case_space_table_names=False, orie
         :param case_space_table_names: boolean - make best guesses how to add spaces and upper case
                                        characters to table names
 
-        :param orient: Indication of expected JSON string format. See pandas.to_json for more details.
+        :param orient: Passed through to pandas.to_json. Default of "split", combined with index=False, writes
+                       a smaller json file.
 
         :param index: boolean - whether or not to write the index.
 
@@ -161,7 +205,11 @@ def write_file(self, pan_dat, json_file_path, case_space_table_names=False, orie
         :param kwargs: additional named arguments to pass to pandas.to_json
 
         :return:
+
+        NB - pandas seems stubbornly unable to inject Infinity into json, but it can read Infinity from
+        json. We work around this with a GUID created flagging string when encountering float("inf"), float(-"inf").
         """
+        infinity_flagging_str = str(uuid.uuid4())[-9:]
         msg = []
         verify(self.pan_dat_factory.good_pan_dat_object(pan_dat, msg.append),
                "pan_dat not a good object for this factory : %s"%"\n".join(msg))
@@ -176,9 +224,9 @@ def write_file(self, pan_dat, json_file_path, case_space_table_names=False, orie
                                  len(set(self.pan_dat_factory.all_tables)) == \
                                  len(set(map(case_space_to_pretty, self.pan_dat_factory.all_tables)))
         rtn = {}
-        from ticdat.pandatfactory import _faster_df_apply
         for t in self.pan_dat_factory.all_tables:
-            df = getattr(pan_dat, t).copy(deep=True).replace(float("inf"), "inf").replace(-float("inf"), "-inf")
+            df = getattr(pan_dat, t).copy(deep=True).replace(float("inf"), infinity_flagging_str).\
+                 replace(-float("inf"), f"-{infinity_flagging_str}")
             for f in df.columns:
                 dt = self.pan_dat_factory.data_types.get(t, {}).get(f, None)
                 if dt and dt.datetime:
@@ -189,9 +237,11 @@ def fixed(row): # this might not always fix things
                         if pd.isnull(row[f]):
                             return None
                         return row[f]
-                    df[f] = _faster_df_apply(df, fixed)
+                    df[f] = faster_df_apply(df, fixed)
             k = case_space_to_pretty(t) if case_space_table_names else t
-            rtn[k] = json.loads(df.to_json(path_or_buf=None, orient=orient, **kwargs))
+            rtn[k] = json.loads(df.to_json(path_or_buf=None, orient=orient, **kwargs).
+                                replace(f'"{infinity_flagging_str}"', "Infinity").replace(
+                                f'"-{infinity_flagging_str}"', "-Infinity"))
             if orient == 'split' and not index:
                 rtn[k].pop("index", None)
         if json_file_path:
@@ -264,12 +314,14 @@ def create_pan_dat(self, dir_path, fill_missing_fields=False, **kwargs):
         missing_fields = {(t, f) for t in rtn for f in all_fields(self.pan_dat_factory, t)
                           if f not in rtn[t].columns}
         if fill_missing_fields:
-            for t,f in missing_fields:
-                rtn[t][f] = self.pan_dat_factory.default_values[t][f]
-        verify(fill_missing_fields or not missing_fields,
+            for t, f in list(missing_fields):
+                if f in self.pan_dat_factory.default_values.get(t, {}):
+                    rtn[t][f] = self.pan_dat_factory.default_values[t][f]
+                    missing_fields.remove((t, f))
+        verify(not missing_fields,
                "The following (table, file_name, field) triplets are missing fields.\n%s" %
                [(t, os.path.basename(tbl_names[t]), f) for t,f in missing_fields])
-        return _clean_pandat_creator(self.pan_dat_factory, rtn, print_missing_tables=True)
+        return _clean_pandat_creator(self.pan_dat_factory, rtn)
 
     def _get_table_names(self, dir_path):
         rtn = {}
@@ -372,9 +424,11 @@ def create_pan_dat(self, db_file_path, con=None, fill_missing_fields=False):
         missing_fields = {(t, f) for t in rtn for f in all_fields(self.pan_dat_factory, t)
                           if f not in rtn[t].columns}
         if fill_missing_fields:
-            for t,f in missing_fields:
-                rtn[t][f] = self.pan_dat_factory.default_values[t][f]
-        verify(fill_missing_fields or not missing_fields,
+            for t, f in list(missing_fields):
+                if f in self.pan_dat_factory.default_values.get(t, {}):
+                    rtn[t][f] = self.pan_dat_factory.default_values[t][f]
+                    missing_fields.remove((t, f))
+        verify(not missing_fields,
                "The following are (table, field) pairs missing from the %s file.\n%s" % (db_file_path, missing_fields))
         missing_tables = sorted(set(self.pan_dat_factory.all_tables).difference(rtn))
         if missing_tables:
@@ -507,12 +561,14 @@ def create_pan_dat(self, xls_file_path, fill_missing_fields=False):
         missing_fields = {(t, f) for t in rtn for f in all_fields(self.pan_dat_factory, t)
                           if f not in rtn[t].columns}
         if fill_missing_fields:
-            for t,f in missing_fields:
-                rtn[t][f] = self.pan_dat_factory.default_values[t][f]
-        verify(fill_missing_fields or not missing_fields,
+            for t, f in list(missing_fields):
+                if f in self.pan_dat_factory.default_values.get(t, {}):
+                    rtn[t][f] = self.pan_dat_factory.default_values[t][f]
+                    missing_fields.remove((t, f))
+        verify(not missing_fields,
                "The following are (table, field) pairs missing from the %s file.\n%s" % (xls_file_path, missing_fields))
         xl.close()
-        rtn = _clean_pandat_creator(self.pan_dat_factory, rtn, print_missing_tables=True)
+        rtn = _clean_pandat_creator(self.pan_dat_factory, rtn)
         if self.pan_dat_factory.xlsx_trailing_empty_rows == "prune":
             from ticdat.pandatfactory import remove_trailing_all_nan
             for t in self.pan_dat_factory.all_tables:
diff --git a/ticdat/ticdatfactory.py b/ticdat/ticdatfactory.py
index 4c625878..6e8b7417 100644
--- a/ticdat/ticdatfactory.py
+++ b/ticdat/ticdatfactory.py
@@ -452,6 +452,32 @@ def enable_foreign_key_links(self):
         :return:
         """
         self._foreign_key_links_enabled[:] = [True]
+    def add_implied_foreign_keys(self):
+        '''
+        Cascades foreign keys downward (i.e. computes implied foreign key relationships)
+        Calling this routine won't change the boolean result of subsequent calls to find_foreign_key_failures,
+        but it might change the size of the find_foreign_key_failures returned dictionary.
+        :return:
+        '''
+        verify(not self._has_been_used,
+                "The foreign keys can't be changed after a TicDatFactory has been used.")
+        def findderivedforeignkey():
+            curFKs = self._foreign_keys_by_native()
+            if not self._complex_fks():
+                for (nativetable, bridgetable), nativebridgemappings in list(self._foreign_keys.items()):
+                    for nb_map in nativebridgemappings :
+                      for bfk in curFKs.get(bridgetable,()):
+                        nativefields = bfk.nativefields()
+                        assert set(nativefields).issubset(self._allFields(bridgetable))
+                        bridgetonative = {bf:nf for nf,bf in nb_map}
+                        foreigntobridge = bfk.foreigntonativemapping()
+                        newnativeft = tuple((bridgetonative[bf], ff) for ff,bf in
+                                             foreigntobridge.items() if bf in bridgetonative)
+                        fkSet = self._foreign_keys[nativetable, bfk.foreign_table]
+                        if newnativeft not in fkSet and self._simple_fk(bfk.foreign_table, newnativeft):
+                            return fkSet.add(newnativeft) or True
+        while findderivedforeignkey():
+            pass
     def add_foreign_key(self, native_table, foreign_table, mappings):
         """
         Adds a foreign key relationship to the schema.  Adding a foreign key doesn't block
@@ -507,24 +533,6 @@ def _complex_fks(self):
     def _trigger_has_been_used(self):
         if self._has_been_used :
             return # idempotent
-        def findderivedforeignkey():
-            # this cascades foreign keys downward (i.e. computes implied foreign key relationships)
-            curFKs = self._foreign_keys_by_native()
-            if not self._complex_fks():
-                for (nativetable, bridgetable), nativebridgemappings in list(self._foreign_keys.items()):
-                    for nb_map in nativebridgemappings :
-                      for bfk in curFKs.get(bridgetable,()):
-                        nativefields = bfk.nativefields()
-                        assert set(nativefields).issubset(self._allFields(bridgetable))
-                        bridgetonative = {bf:nf for nf,bf in nb_map}
-                        foreigntobridge = bfk.foreigntonativemapping()
-                        newnativeft = tuple((bridgetonative[bf], ff) for ff,bf in
-                                             foreigntobridge.items() if bf in bridgetonative)
-                        fkSet = self._foreign_keys[nativetable, bfk.foreign_table]
-                        if newnativeft not in fkSet and self._simple_fk(bfk.foreign_table, newnativeft):
-                            return fkSet.add(newnativeft) or True
-        while findderivedforeignkey():
-            pass
         for (nativetable, foreigntable), nativeforeignmappings in self._foreign_keys.items():
             nativeFieldsSet = frozenset(frozenset(_[0] for _ in nfm) for nfm in nativeforeignmappings
                                         if self._simple_fk(foreigntable, nfm)) # ignore for complex
@@ -708,6 +716,7 @@ def __init__(self, **init_tables):
                 superself._trigger_has_been_used()
                 self._all_data_dicts = []
                 self._made_foreign_links = False
+                lens = {t: l for t, v in init_tables.items() for l in [utils.safe_apply(len)(v)] if l is not None}
                 for t in init_tables :
                     verify(t in superself.all_tables, "Unexpected table name %s"%t)
                     if t in superself.generic_tables:
@@ -722,14 +731,20 @@ def __init__(self, **init_tables):
                         v = DataFrame(v)
                         v.rename(columns = {v.columns[0] : superself.data_fields[t][0]}, inplace=True)
                     if DataFrame and isinstance(v, DataFrame):
-                      row_dict = lambda r : {df:r[df] for df in superself.data_fields.get(t, ())}
+                      apply = utils.faster_df_apply
                       setattr(self, t, ticdattablefactory(self._all_data_dicts, t)())
-                      if superself.primary_key_fields.get(t) :
-                          def add_row(r):
-                              getattr(self, t)[r.name] = row_dict(r)
-                          v.apply(add_row, axis=1)
+                      _rd_ = lambda row_dict: {k: row_dict[k] for k in superself.data_fields.get(t, [])}
+                      if superself.primary_key_fields.get(t):
+                          if not set(superself.primary_key_fields[t]).issubset(v.columns):
+                              v = v.reset_index(drop=False)
+                          key_maker = (lambda rd: rd[superself.primary_key_fields[t][0]]) \
+                                      if len(superself.primary_key_fields[t]) == 1 else  \
+                                      (lambda rd: tuple(rd[_] for _ in superself.primary_key_fields[t]))
+                          def add_row(row_dict):
+                              getattr(self, t)[key_maker(row_dict)] = _rd_(row_dict)
+                          apply(v, add_row)
                       else :
-                          v.apply(lambda r : getattr(self, t).append(row_dict(r)), axis=1)
+                          apply(v, lambda row_dict: getattr(self, t).append(_rd_(row_dict)))
                     elif superself.primary_key_fields.get(t) and not utils.dictish(v):
                          pklen = len(superself.primary_key_fields[t])
                          def handle_row_dict(r):
@@ -767,6 +782,8 @@ def handle_row_dict(r):
                         setattr(self, t, ticdattablefactory(self._all_data_dicts, t)())
                 if init_tables :
                     self._try_make_foreign_links()
+                if any(v > (l or 0) for k, v in lens.items() for l in [utils.safe_apply(len)(getattr(self, k))]):
+                    print("--> Warning: Some rows have been lost due to duplicate rows passed to TicDat.__init__")
             def _try_make_foreign_links(self):
                 if not superself._foreign_key_links_enabled:
                     return
@@ -928,7 +945,7 @@ def _infinity_flag_write_cell(self, t, f, x):
         """
         if t == "parameters" and self._parameters: # SPEED IS IMPORTANT
             # I will assume a parameters table without parameters specification is just a naive developer
-            return None if x is None or (utils.pd and utils.pd.isnull(x)) else str(x)
+            return None if x is None or (utils.pd and utils.pd.isnull(x)) else x
         if self.infinity_io_flag is None and (self._none_as_infinity_bias(t, f) or float("nan"))*float("inf") == x:
             return None
         if utils.numericish(self.infinity_io_flag) and utils.numericish(x):
@@ -1080,12 +1097,13 @@ def good_tic_dat_table(self, data_table, table_name, bad_message_handler = lambd
                 bad_message_handler("%s has the string 'name' as part of a multi-field "%table_name +
                                     "primary key and thus cannot be populated with a DataFrame")
                 return False
-            if pks and (pks != utils.safe_apply(lambda : tuple(data_table.index.names))()) :
-                bad_message_handler("Could not find a pandas index matching the primary key for %s"%table_name)
-                return False
-            if not set(data_table.columns).issuperset(self.data_fields.get(table_name, ())) :
+            if not set(data_table.columns).issuperset(self.data_fields.get(table_name, ())):
                 bad_message_handler("Could not find pandas columns for all the data fields for %s"%table_name)
                 return False
+            if pks and (pks != utils.safe_apply(lambda : tuple(data_table.index.names))()) and \
+                not set(data_table.columns).issuperset(pks):
+                bad_message_handler("Unable to find pandas column nor index matching the primary key for %s"%table_name)
+                return False
             return True
         if self.primary_key_fields.get(table_name):
             if row_checking == "strict" and  not utils.dictish(data_table):
@@ -1219,14 +1237,22 @@ def containerize(_r) :
     def clone(self, table_restrictions=None, clone_factory=None):
         """
         clones the TicDatFactory
+
         :param table_restrictions : if None, then argument is ignored. Otherwise, a container listing the
                                     tables to keep in the clone. Tables outside table_restrictions are removed from
                                     the clone.
+
         :param clone_factory : optional. Defaults to TicDatFactory. Can also be PanDatFactory. Can also be a function,
                                in which case it should behave similarly to create_from_full_schema.
+                               If clone_factory=PanDatFactory, the row predicates that use predicate_kwargs_maker
+                               won't be copied over.
+
         :return: a clone of the TicDatFactory. Returned object will based on clone_factory, if provided.
+
         """
         clone_factory = clone_factory or TicDatFactory
+        from ticdat import PanDatFactory
+        no_copy_predicate_kwargs_maker = clone_factory == PanDatFactory
         if hasattr(clone_factory, "create_from_full_schema"):
             clone_factory = clone_factory.create_from_full_schema
         full_schema = utils.clone_a_anchillary_info_schema(self.schema(include_ancillary_info=True), table_restrictions)
@@ -1236,9 +1262,10 @@ def clone(self, table_restrictions=None, clone_factory=None):
         for tbl, row_predicates in self._data_row_predicates.items():
             if table_restrictions is None or tbl in table_restrictions:
                 for pn, rpi in row_predicates.items():
-                    rtn.add_data_row_predicate(tbl, predicate=rpi.predicate, predicate_name=pn,
-                                               predicate_kwargs_maker=rpi.predicate_kwargs_maker,
-                                               predicate_failure_response=rpi.predicate_failure_response)
+                    if not (rpi.predicate_kwargs_maker and no_copy_predicate_kwargs_maker):
+                        rtn.add_data_row_predicate(tbl, predicate=rpi.predicate, predicate_name=pn,
+                                                   predicate_kwargs_maker=rpi.predicate_kwargs_maker,
+                                                   predicate_failure_response=rpi.predicate_failure_response)
         rtn.enable_foreign_key_links() if self._foreign_key_links_enabled else None
         return rtn
     def copy_tic_dat(self, tic_dat, freeze_it = False):
@@ -1755,7 +1782,7 @@ def find_data_row_failures(self, tic_dat, exception_handling="__debug__", max_fa
               "Unhandled": Exceptions resulting from calling a row predicate (or a predicate_kwargs_maker) will not be
                            handled by data_row_failures.
               "__debug__": Since "Handled as Failure" makes more sense for production runs and "Unhandled" makes more
-                           sense for debugging, this value will use the latter if __debug__ is True and the former
+                           sense for debugging, this option will use the latter if __debug__ is True and the former
                            otherwise. See -o and __debug__ in Python documentation for more details.
 
         :param max_failures: number. An upper limit on the number of failures to find. Will short circuit and return
@@ -1893,9 +1920,9 @@ def obfusimplify(self, tic_dat, table_prepends = utils.FrozenDict(), skip_tables
         """
         msg  = []
         verify(self.good_tic_dat_object(tic_dat, msg.append),
-               "tic_dat not a good object for this factory : %s"%"\n".join(msg))
+               "tic_dat not a good object for this factory : %s" %"\n".join(msg))
         verify(not self._complex_fks(), ("complex foreign key between %s and %s prevents " +
-                                         "obfusimplify")%((self._complex_fks() or [(None,)*3])[0][:2]))
+                                         "obfusimplify") % ((self._complex_fks() or [(None,) * 3])[0][:2]))
         verify({fk.cardinality for fk in self.foreign_keys}.issubset({"many-to-one", "one-to-one"}),
                "many-to-many and one-to-many foreign keys are not currently supported for obfusimplify")
         verify(not self.find_foreign_key_failures(tic_dat),
@@ -1903,11 +1930,11 @@ def obfusimplify(self, tic_dat, table_prepends = utils.FrozenDict(), skip_tables
         verify(not self.generator_tables, "Cannot obfusimplify a tic_dat that uses generators")
         verify(not set(table_prepends).intersection(skip_tables),
                "Can't specify a table prepend for an entity that you're skipping")
-        verify(self._has_been_used,
-               "The cascading foreign keys won't necessarily be present until the factory is used")
+        better_self = self.clone()
+        better_self.add_implied_foreign_keys()
 
-        entity_tables = {t for t,v in self.primary_key_fields.items() if len(v) == 1}
-        foreign_keys_by_native = self._foreign_keys_by_native()
+        entity_tables = {t for t,v in better_self.primary_key_fields.items() if len(v) == 1}
+        foreign_keys_by_native = better_self._foreign_keys_by_native()
         # if a native table is one-to-one with a foreign table, it isn't an entity table
         for nt in entity_tables.intersection(foreign_keys_by_native):
             if any(ft.cardinality == "one-to-one" for ft in foreign_keys_by_native[nt]):
@@ -1917,8 +1944,8 @@ def obfusimplify(self, tic_dat, table_prepends = utils.FrozenDict(), skip_tables
         entity_tables = entity_tables.difference(skip_tables)
 
         for k,v in table_prepends.items():
-            verify(k in self.all_tables, "%s is not a table name")
-            verify(len(self.primary_key_fields.get(k, ())) ==1, "%s does not have a single primary key field"%k)
+            verify(k in better_self.all_tables, "%s is not a table name")
+            verify(len(better_self.primary_key_fields.get(k, ())) == 1, f"{k} does not have a single primary key field")
             verify(k in entity_tables, "%s is not an entity table due to child foreign key relationship"%k)
             verify(utils.stringish(v) and set(v).issubset(uppercase) and not v.endswith("I"),
                    "Your table_prepend string %s is not an all uppercase string ending in a letter other than I")
@@ -1949,7 +1976,7 @@ def getname():
             for i,k in enumerate(sorted(getattr(tic_dat, t))) :
                 reverse_renamings[t, k] = "%s%s"%(table_prepends[t],i+1)
         foreign_keys = {}
-        for fk in self.foreign_keys:
+        for fk in better_self.foreign_keys:
             nt = fk.native_table
             if fk.foreign_table in table_prepends:
                 foreign_keys = utils.dict_overlay(foreign_keys,
@@ -1957,25 +1984,25 @@ def getname():
                                   fk.nativetoforeignmapping().items()})
         # remember -- we've used this factory so any cascading foreign keys are present
         rtn_dict  = clt.defaultdict(dict)
-        for t in self.all_tables:
+        for t in better_self.all_tables:
             read_table = getattr(tic_dat, t)
             def fix_all_row(all_row):
                 return {k: reverse_renamings[foreign_keys[t, k], v] if (t,k) in foreign_keys else v
                            for k,v in all_row.items()}
             if dictish(read_table):
                 for pk, data_row in read_table.items():
-                    if len(self.primary_key_fields.get(t, ())) == 1:
-                        pkf = self.primary_key_fields[t][0]
+                    if len(better_self.primary_key_fields.get(t, ())) == 1:
+                        pkf = better_self.primary_key_fields[t][0]
                         if (t,pkf) in foreign_keys:
                             new_pk = reverse_renamings[foreign_keys[t,pkf], pk]
                         else:
                             new_pk = reverse_renamings.get((t, pk), pk)
                         rtn_dict[t][new_pk] = fix_all_row(data_row)
                     else :
-                        assert containerish(pk) and len(pk) == len(self.primary_key_fields[t])
+                        assert containerish(pk) and len(pk) == len(better_self.primary_key_fields[t])
                         new_row = fix_all_row(dict(data_row, **{pkf: pkv for pkf, pkv in
-                                                    zip(self.primary_key_fields[t], pk)}))
-                        new_pk = tuple(new_row[_] for _ in self.primary_key_fields[t])
+                                                                zip(better_self.primary_key_fields[t], pk)}))
+                        new_pk = tuple(new_row[_] for _ in better_self.primary_key_fields[t])
                         rtn_dict[t][new_pk] = {k:new_row[k] for k in data_row}
             else :
                 rtn_dict[t] = []
@@ -1984,9 +2011,9 @@ def fix_all_row(all_row):
 
         RtnType = namedtuple("ObfusimplifyResults", "copy renamings")
 
-        rtn = RtnType(self.freeze_me(self.TicDat(**rtn_dict)) if freeze_it else self.TicDat(**rtn_dict),
+        rtn = RtnType(better_self.freeze_me(better_self.TicDat(**rtn_dict)) if freeze_it else better_self.TicDat(**rtn_dict),
                       {v:k for k,v in reverse_renamings.items()})
-        assert not self.find_foreign_key_failures(rtn.copy)
+        assert not better_self.find_foreign_key_failures(rtn.copy)
         assert len(rtn.renamings) == len(reverse_renamings)
         return rtn
 
diff --git a/ticdat/utils.py b/ticdat/utils.py
index 9a47e9bf..bf68b38f 100644
--- a/ticdat/utils.py
+++ b/ticdat/utils.py
@@ -29,6 +29,46 @@
     drm = None
 import inspect
 
+def faster_df_apply(df, func, trip_wire_check=None):
+    """
+    pandas.DataFrame.apply is rarely used because it is slow. It is slow because it creates a Series for each row
+    of the DataFrame, and passes this Series to the function. faster_df_apply creates a dict for each row of the
+    DataFrame instead, and as a result is **much** faster.
+
+    See https://bit.ly/3xnLFld.
+
+    It's certainly possible newer versions of pandas will implement a more performant DataFrame.apply. The broader
+    point is, row-wise apply should not be discarded wholesale for performance reasons, as DataFrame.itertuples
+    is reasonably fast
+
+    :param df: a DataFrame
+
+    :param func: a function to apply to each row of the DataFrame. The function should accept a fieldname->data
+                 dictionary as argument. func will be applied to each row of the DataFrame
+
+    :param trip_wire_check: optional. If provided, a function that will be passed each result returned by func.
+                                      trip_wire_check can either return falsey, or a replacement to func to be applied
+                                      to the remainder of the DataFrame
+
+    :return: a pandas Series with the same index as df and the values of calling func on each row dict.
+    """
+    verify(DataFrame and isinstance(df, DataFrame), "df argument needs to be a DataFrame")
+    verify(callable(func), "func needs to be a function")
+    verify(not trip_wire_check or callable(trip_wire_check), "trip_wire_check needs to None, or a function")
+    cols = list(df.columns)
+    data, index = [], []
+    for row in df.itertuples(index=True):
+        row_dict = {f:v for f,v in zip(cols, row[1:])}
+        data.append(func(row_dict))
+        index.append(row[0])
+        if trip_wire_check:
+            new_func = trip_wire_check(data[-1])
+            if new_func:
+                func = new_func
+                trip_wire_check = None
+    # will default to float for empty Series, like original pandas
+    return pd.Series(data, index=index, **({"dtype": numpy.float64} if not data else {}))
+
 def dat_restricted(table_list):
     '''
     Decorator factory used to decorate action functions (or solve function) to restrict the access to the
@@ -1072,8 +1112,8 @@ def notify_progress(self, progress_data):
 EPSILON = 1e-05
 
 def per_error(x1, x2) :
-    x1 = float(x1)
-    x2 = float(x2)
+    x1 = float(x1) if numericish(x1) else x1
+    x2 = float(x2) if numericish(x2) else x2
     if (x1 < 0) and (x2 < 0) :
         return per_error(-x1, -x2)
     if x1 == float("inf") :
diff --git a/ticdat/xls.py b/ticdat/xls.py
index ee4e5888..6be01f48 100644
--- a/ticdat/xls.py
+++ b/ticdat/xls.py
@@ -453,7 +453,7 @@ def _xlsx_write(self, tic_dat, file_path, tbl_name_mapping):
         def clean_for_write(t, f, x):
             if self.tic_dat_factory.infinity_io_flag != "N/A" or \
                (t == "parameters" and self.tic_dat_factory.parameters):
-                return self.tic_dat_factory._infinity_flag_write_cell(t, f, x)
+                x = self.tic_dat_factory._infinity_flag_write_cell(t, f, x)
             if x in [float("inf"), -float("inf")] or isinstance(x, datetime.datetime):
                 return str(x)
             return x
@@ -468,7 +468,8 @@ def clean_for_write(t, f, x):
                 for row_ind, (p_key, data) in enumerate(_t.items()) :
                     for field_ind, cell in enumerate( (p_key if containerish(p_key) else (p_key,)) +
                                         tuple(data[_f] for _f in tdf.data_fields.get(t, ()))):
-                        sheet.write(row_ind+1, field_ind, clean_for_write(t, all_flds[field_ind], cell))
+                        write_cell =  clean_for_write(t, all_flds[field_ind], cell)
+                        sheet.write(row_ind+1, field_ind, write_cell)
             else :
                 for row_ind, data in enumerate(_t if containerish(_t) else _t()) :
                     for field_ind, cell in enumerate(tuple(data[_f] for _f in tdf.data_fields[t])) :


Test patch:
diff --git a/ticdat/testing/test_pgtd.py b/ticdat/testing/test_pgtd.py
index 2cf70269..5e771ad0 100644
--- a/ticdat/testing/test_pgtd.py
+++ b/ticdat/testing/test_pgtd.py
@@ -191,12 +191,9 @@ def test_issue_68_pd(self):
         pgtf.write_data(pan_dat, self.engine, test_schema)
         pg_pan_dat = pgtf.create_pan_dat(self.engine, test_schema)
         self.assertTrue(pdf._same_data(pan_dat, pg_pan_dat))
-        from ticdat.pandatfactory import _faster_df_apply
-        pan_dat.categories["Max Nutrition"] = _faster_df_apply(pan_dat.categories,
+        pan_dat.categories["Max Nutrition"] = utils.faster_df_apply(pan_dat.categories,
                                                                lambda row: numpy.int64(row["Max Nutrition"]))
-        pan_dat.foods["Cost"] = _faster_df_apply(pan_dat.foods, lambda row: numpy.float64(row["Cost"]))
-        from framework_utils.helper_utils import memo
-        memo(pan_dat)
+        pan_dat.foods["Cost"] = utils.faster_df_apply(pan_dat.foods, lambda row: numpy.float64(row["Cost"]))
         pgtf.write_data(pan_dat, self.engine, test_schema)
         pg_pan_dat = pgtf.create_pan_dat(self.engine, test_schema)
         self.assertTrue(pdf._same_data(pan_dat, pg_pan_dat))
diff --git a/ticdat/testing/testjson.py b/ticdat/testing/testjson.py
index 50c173d4..224c43b8 100644
--- a/ticdat/testing/testjson.py
+++ b/ticdat/testing/testjson.py
@@ -1,10 +1,13 @@
 import os
 import shutil
 from ticdat.ticdatfactory import TicDatFactory
+from ticdat.pandatfactory import PanDatFactory
 from ticdat.testing.ticdattestutils import dietData, dietSchema, netflowData, dietSchemaWeirdCase
 from ticdat.testing.ticdattestutils import  netflowSchema, firesException, copyDataDietWeirdCase
 from ticdat.testing.ticdattestutils import sillyMeData, sillyMeSchema, sillyMeDataTwoTables, fail_to_debugger
 from ticdat.testing.ticdattestutils import makeCleanDir, dietSchemaWeirdCase2, copyDataDietWeirdCase2, makeCleanPath
+from ticdat.testing.ticdattestutils import flagged_as_run_alone
+
 import unittest
 from ticdat.jsontd import _can_unit_test, json
 import datetime
@@ -36,32 +39,34 @@ def firesException(self, f, troubleshoot=False):
     def testDiet(self):
         if not self.can_run:
             return
-        for verbose in [True, False]:
+        for kwargs in [{"verbose": True}, {"verbose": False}, {"to_pandas": True}]:
             tdf = TicDatFactory(**dietSchema())
-            ticDat = tdf.freeze_me(tdf.TicDat(**{t:getattr(dietData(),t) for t in tdf.primary_key_fields}))
+            ticDat = tdf.TicDat(**{t:getattr(dietData(),t) for t in tdf.primary_key_fields})
+            ticDat.categories["fat"]["minNutrition"] = -float("inf") # violates integrity but better testing
+            ticDat = tdf.freeze_me(ticDat)
             writePath = os.path.join(makeCleanDir(os.path.join(_scratchDir, "diet")), "file.json")
-            tdf.json.write_file(ticDat, writePath, verbose=verbose)
+            tdf.json.write_file(ticDat, writePath, **kwargs)
             self.assertFalse(tdf.json.find_duplicates(writePath))
             jsonTicDat = tdf.json.create_tic_dat(writePath)
-            self.assertTrue(tdf._same_data(ticDat, jsonTicDat))
+            self.assertTrue(tdf._same_data(ticDat, jsonTicDat, epsilon=1e-5))
 
             def change() :
                 jsonTicDat.categories["calories"]["minNutrition"]=12
             self.assertFalse(firesException(change))
-            self.assertFalse(tdf._same_data(ticDat, jsonTicDat))
+            self.assertFalse(tdf._same_data(ticDat, jsonTicDat, epsilon=1e-5))
             jsonTicDat = tdf.json.create_tic_dat(writePath, freeze_it=True)
             self.assertTrue(firesException(change))
-            self.assertTrue(tdf._same_data(ticDat, jsonTicDat))
+            self.assertTrue(tdf._same_data(ticDat, jsonTicDat, epsilon=1e-5))
 
         tdf2 = TicDatFactory(**dietSchemaWeirdCase())
         dat2 = copyDataDietWeirdCase(ticDat)
-        tdf2.json.write_file(dat2, writePath, allow_overwrite=True, verbose=verbose)
+        tdf2.json.write_file(dat2, writePath, allow_overwrite=True)
         jsonTicDat2 = tdf.json.create_tic_dat(writePath, freeze_it=True)
         self.assertTrue(tdf._same_data(ticDat, jsonTicDat2))
 
         tdf3 = TicDatFactory(**dietSchemaWeirdCase2())
         dat3 = copyDataDietWeirdCase2(ticDat)
-        tdf3.json.write_file(dat3, writePath, allow_overwrite=True, verbose=verbose)
+        tdf3.json.write_file(dat3, writePath, allow_overwrite=True)
         with open(writePath, "r") as f:
             jdict = json.load(f)
         jdict["nutrition quantities"] = jdict["nutrition_quantities"]
@@ -79,12 +84,12 @@ def testSillyTwoTables(self):
         if not self.can_run:
             return
 
-        for verbose in [True, False]:
+        for kwargs in [{"verbose": True}, {"verbose": False}, {"to_pandas": True}]:
 
             tdf = TicDatFactory(**sillyMeSchema())
             ticDat = tdf.TicDat(**sillyMeDataTwoTables())
             writePath = os.path.join(makeCleanDir(os.path.join(_scratchDir, "sillyTwoTables")), "file.json")
-            tdf.json.write_file(ticDat, writePath, verbose=verbose)
+            tdf.json.write_file(ticDat, writePath, **kwargs)
             self.assertFalse(tdf.json.find_duplicates(writePath))
             jsonTicDat = tdf.json.create_tic_dat(writePath)
             self.assertTrue(tdf._same_data(ticDat, jsonTicDat))
@@ -152,7 +157,7 @@ def testDups(self):
                                 for t in tdf.all_tables})
             writePath = os.path.join(makeCleanDir(os.path.join(_scratchDir, "dups")), "file.json")
             tdf2.json.write_file(td, writePath, **kwargs)
-            dups = tdf.json.find_duplicates(writePath, from_pandas=kwargs.get("to_pandas", False))
+            dups = tdf.json.find_duplicates(writePath)
             self.assertTrue(dups == {'three': {(1, 2, 2): 2}, 'two': {(1, 2): 3}, 'one': {1: 3, 2: 2}})
 
     def testSilly(self):
@@ -228,6 +233,7 @@ def testDietWithInfFlagging(self):
         dat_1 = tdf.json.create_tic_dat(file_one)
         self.assertFalse(tdf._same_data(dat, dat_1))
 
+
     def test_parameters(self):
         path = os.path.join(_scratchDir, "parameters.json")
         tdf = TicDatFactory(parameters=[["Key"], ["Value"]])
@@ -237,6 +243,15 @@ def test_parameters(self):
         tdf.json.write_file(dat, path)
         dat_ = tdf.json.create_tic_dat(path)
         self.assertTrue(tdf._same_data(dat, dat_))
+        tdf2 = TicDatFactory(parameters=[["Key"], ["Value"]])
+        pdf = PanDatFactory(parameters=[["Key"], ["Value"]])
+        dat = tdf.TicDat(parameters = [["Something", 2]])
+        dat_ = tdf2.json.create_tic_dat(tdf.json.write_file(dat, ""))
+        self.assertTrue(tdf._same_data(dat, dat_))
+        dat_2 = pdf.copy_to_tic_dat(pdf.json.create_pan_dat(tdf.json.write_file(dat, "", to_pandas=True)))
+        self.assertTrue(tdf._same_data(dat, dat_2))
+
+
 
     def testDateTime(self):
         tdf = TicDatFactory(table_with_stuffs = [["field one"], ["field two"]],
diff --git a/ticdat/testing/testpandas.py b/ticdat/testing/testpandas.py
index ae56b507..2b35bd15 100644
--- a/ticdat/testing/testpandas.py
+++ b/ticdat/testing/testpandas.py
@@ -121,10 +121,8 @@ def testNetflow(self):
                             for src, r in oldDat.nodes["Boston"].arcs_destination.items()))
         ticDat = tdf.copy_to_pandas(oldDat, drop_pk_columns=True)
         rebornTicDat = tdf.TicDat(**{t:getattr(ticDat, t) for t in tdf.all_tables})
-        # because we have single pk field tables, dropping the pk columns is probelmatic
-        self.assertFalse(tdf._same_data(rebornTicDat, oldDat))
+        self.assertTrue(tdf._same_data(rebornTicDat, oldDat))
 
-        # but with the default argument all is well
         ticDat = tdf.copy_to_pandas(oldDat)
         rebornTicDat = tdf.TicDat(**{t:getattr(ticDat, t) for t in tdf.all_tables})
         self.assertTrue(tdf._same_data(rebornTicDat, oldDat))
diff --git a/ticdat/testing/testpandat_io.py b/ticdat/testing/testpandat_io.py
index dc088fc2..5e5e88ac 100644
--- a/ticdat/testing/testpandat_io.py
+++ b/ticdat/testing/testpandat_io.py
@@ -131,6 +131,16 @@ def testXlsSimple(self):
         xlsPanDat = pdf2.xls.create_pan_dat(filePath)
         self.assertTrue(pdf._same_data(panDat, xlsPanDat))
 
+    def testReadingJsonFromTDF(self):
+        tdf = TicDatFactory(**dietSchema())
+        pdf = PanDatFactory(**dietSchema())
+        ticDat = tdf.TicDat(**{t: getattr(dietData(), t) for t in tdf.primary_key_fields})
+        ticDat.categories["fat"]["minNutrition"] = -float("inf")  # violates integrity but better testing
+        writePath = os.path.join(_scratchDir, "read_from_TDF.json")
+        tdf.json.write_file(ticDat, writePath)
+        dat2 = pdf.copy_to_tic_dat(pdf.json.create_pan_dat(writePath))
+        self.assertTrue(tdf._same_data(ticDat, dat2, epsilon=1e-5))
+
     def testDietWithInfFlagging(self):
         diet_pdf = PanDatFactory(**dietSchema())
         addDietDataTypes(diet_pdf)
@@ -140,10 +150,11 @@ def testDietWithInfFlagging(self):
         core_path = os.path.join(_scratchDir, "diet_with_inf_flagging")
         diet_pdf.sql.write_file(dat, core_path+".db")
         diet_pdf.csv.write_directory(dat, core_path+"_csv")
-        diet_pdf.json.write_file(dat, core_path+".json")
+        diet_pdf.json.write_file_pd(dat, core_path+".json")
+        diet_pdf.json.write_file(dat, core_path+"2.json")
         diet_pdf.xls.write_file(dat, core_path+".xlsx")
         for attr, f in [["sql", core_path+".db"], ["csv", core_path+"_csv"], ["json", core_path+".json"],
-                        ["xls", core_path+".xlsx"]]:
+                        ["json", core_path+"2.json"], ["xls", core_path+".xlsx"]]:
             dat_1 = getattr(diet_pdf, attr).create_pan_dat(f)
             self.assertTrue(diet_pdf._same_data(dat, dat_1, epsilon=1e-5))
             pdf = diet_pdf.clone()
@@ -162,9 +173,11 @@ def test_missing_tables(self):
         pdf_1 = PanDatFactory(this = [["Something"],["Another"]])
         pdf_2 = PanDatFactory(**dict(pdf_1.schema(), that=[["What", "Ever"],[]]))
         dat = pdf_1.PanDat(this={"Something": ["a", "b", "c"], "Another": [2, 3, 5]})
-        for attr, path in [["sql", core_path+".db"], ["csv", core_path+"_csv"], ["json", core_path+".json"],
-                           ["xls", core_path+".xlsx"]]:
-            func = "write_directory" if attr == "csv" else "write_file"
+        for attr, path, func in [["sql", core_path+".db", "write_file"],
+                           ["csv", core_path+"_csv", "write_directory"],
+                           ["json", core_path+".json", "write_file_pd"],
+                           ["json", core_path + "2.json", "write_file"],
+                           ["xls", core_path+".xlsx", "write_file"]]:
             getattr(getattr(pdf_1, attr), func)(dat, path)
             dat_1 = getattr(pdf_2, attr).create_pan_dat(path)
             self.assertTrue(pdf_1._same_data(dat, dat_1))
@@ -185,9 +198,11 @@ def test_datetime(self):
         dat = TicDatFactory(**pdf.schema()).copy_to_pandas(dat, drop_pk_columns=False)
         self.assertFalse(pdf.find_data_type_failures(dat) or pdf.find_data_row_failures(dat))
 
-        for attr, path in [["csv", core_path+"_csv"], ["xls", core_path+".xlsx"], ["sql", core_path+".db"],
-                           ["json", core_path+".json"]]:
-            func = "write_directory" if attr == "csv" else "write_file"
+        for attr, path, func in [["sql", core_path+".db", "write_file"],
+                           ["csv", core_path+"_csv", "write_directory"],
+                           ["json", core_path+".json", "write_file_pd"],
+                           ["json", core_path + "2.json", "write_file"],
+                           ["xls", core_path+".xlsx", "write_file"]]:
             getattr(getattr(pdf, attr), func)(dat, path)
             dat_1 = getattr(pdf, attr).create_pan_dat(path)
             self.assertFalse(pdf._same_data(dat, dat_1))
@@ -222,18 +237,22 @@ def test_parameters(self):
         pdf.add_parameter("Different", 'boo', strings_allowed='*', number_allowed=False)
         dat = TicDatFactory(**pdf.schema()).TicDat(parameters = [["Something",float("inf")], ["Different", "inf"]])
         dat = TicDatFactory(**pdf.schema()).copy_to_pandas(dat, drop_pk_columns=False)
-        for attr, path in [["sql", core_path+".db"], ["csv", core_path+"_csv"], ["json", core_path+".json"],
-                           ["xls", core_path+".xlsx"]]:
-            func = "write_directory" if attr == "csv" else "write_file"
+        for attr, path, func in [["sql", core_path+".db", "write_file"],
+                           ["csv", core_path+"_csv", "write_directory"],
+                           ["json", core_path+".json", "write_file_pd"],
+                           ["json", core_path + "2.json", "write_file"],
+                           ["xls", core_path+".xlsx", "write_file"]]:
             getattr(getattr(pdf, attr), func)(dat, path)
             dat_1 = getattr(pdf, attr).create_pan_dat(path)
             self.assertTrue(pdf._same_data(dat, dat_1))
         core_path = os.path.join(_scratchDir, "parameters_two")
         dat = TicDatFactory(**pdf.schema()).TicDat(parameters = [["Something",float("inf")], ["Different", "05701"]])
         dat = TicDatFactory(**pdf.schema()).copy_to_pandas(dat, drop_pk_columns=False)
-        for attr, path in [["sql", core_path+".db"], ["csv", core_path+"_csv"], ["xls", core_path+".xlsx"],
-                           ["json", core_path + ".json"]]:
-            func = "write_directory" if attr == "csv" else "write_file"
+        for attr, path, func in [["sql", core_path+".db", "write_file"],
+                           ["csv", core_path+"_csv", "write_directory"],
+                           ["json", core_path+".json", "write_file_pd"],
+                           ["json", core_path + "2.json", "write_file"],
+                           ["xls", core_path+".xlsx", "write_file"]]:
             getattr(getattr(pdf, attr), func)(dat, path)
             dat_1 = getattr(pdf, attr).create_pan_dat(path)
             self.assertTrue(pdf._same_data(dat, dat_1))
@@ -248,9 +267,11 @@ def make_dat(l):
             return tdf.copy_to_pandas(tdf.TicDat(table = l), drop_pk_columns=False)
         dat = make_dat([[None, 100], [200, 109], [0, 300], [300, None], [400, 0]])
         core_path = os.path.join(_scratchDir, "non_inf_flagging")
-        for attr, path in [["sql", core_path+".db"], ["csv", core_path+"_csv"], ["json", core_path+".json"],
-                           ["xls", core_path+".xlsx"]]:
-            func = "write_directory" if attr == "csv" else "write_file"
+        for attr, path, func in [["sql", core_path+".db", "write_file"],
+                                 ["csv", core_path+"_csv", "write_directory"],
+                                 ["json", core_path+".json", "write_file_pd"],
+                                 ["json", core_path + "2.json", "write_file"],
+                                 ["xls", core_path+".xlsx", "write_file"]]:
             getattr(getattr(pdf, attr), func)(dat, path)
             dat_1 = getattr(pdf, attr).create_pan_dat(path)
             _ = PanDatFactory(table=[[], ["field one", "field two"]])
@@ -319,6 +340,8 @@ def testDefaultAdd(self):
         self.assertTrue("missing" in ex and "extra" in ex)
         ex = self.firesException(lambda : pdf2.csv.create_pan_dat(csvDirPath))
         self.assertTrue("missing" in ex and "extra" in ex)
+        ex = self.firesException(lambda : pdf2.json.create_pan_dat(pdf.json.write_file_pd(panDat, "")))
+        self.assertTrue("missing" in ex and "extra" in ex)
         ex = self.firesException(lambda : pdf2.json.create_pan_dat(pdf.json.write_file(panDat, "")))
         self.assertTrue("missing" in ex and "extra" in ex)
 
@@ -337,7 +360,7 @@ def testDefaultAdd(self):
         panDat2.foods.drop("extra", axis=1, inplace=True)
         self.assertTrue(pdf._same_data(panDat, panDat2))
 
-        panDat2 = pdf2.json.create_pan_dat(pdf.json.write_file(panDat, ""), fill_missing_fields=True)
+        panDat2 = pdf2.json.create_pan_dat(pdf.json.write_file_pd(panDat, ""), fill_missing_fields=True)
         self.assertTrue(set(panDat2.foods["extra"]) == {0})
         panDat2.foods.drop("extra", axis=1, inplace=True)
         self.assertTrue(pdf._same_data(panDat, panDat2, epsilon=1e-5))
@@ -359,11 +382,17 @@ def testDefaultAdd(self):
         panDat3.foods.drop("extra", axis=1, inplace=True)
         self.assertTrue(pdf._same_data(panDat, panDat3))
 
-        panDat3 = pdf3.json.create_pan_dat(pdf.json.write_file(panDat, ""), fill_missing_fields=True)
+        panDat3 = pdf3.json.create_pan_dat(pdf.json.write_file_pd(panDat, ""), fill_missing_fields=True)
         self.assertTrue(set(panDat3.foods["extra"]) == {13})
         panDat3.foods.drop("extra", axis=1, inplace=True)
         self.assertTrue(pdf._same_data(panDat, panDat3, epsilon=1e-5))
 
+        ex = []
+        try:
+            pdf3.json.create_pan_dat(pdf.json.write_file(panDat, ""), fill_missing_fields=True)
+        except utils.TicDatError as e:
+            ex.append(e)
+        self.assertTrue(ex)
 
     def testSqlSimple(self):
         if not self.can_run:
@@ -551,43 +580,44 @@ def testCsvSpacey(self):
     def testJsonSimple(self):
         if not self.can_run:
             return
-        tdf = TicDatFactory(**dietSchema())
-        pdf = PanDatFactory(**dietSchema())
-        ticDat = tdf.freeze_me(tdf.TicDat(**{t:getattr(dietData(),t) for t in tdf.primary_key_fields}))
-        panDat = pan_dat_maker(dietSchema(), ticDat)
-        filePath = os.path.join(_scratchDir, "diet.json")
-        pdf.json.write_file(panDat, filePath)
-        panDat2 = pdf.json.create_pan_dat(filePath)
-        self.assertTrue(pdf._same_data(panDat, panDat2, epsilon=1e-5))
-        pdf2 = PanDatFactory(**{t:'*' for t in pdf.all_tables})
-        pdf2.json.write_file(panDat, filePath)
-        panDat2 = pdf2.json.create_pan_dat(filePath)
-        self.assertTrue(pdf._same_data(panDat, panDat2, epsilon=1e-5))
-
-        re_fielded_schema = {"categories" : (("name",),["maxNutrition", "minNutrition"]),
-     "foods" :[["name"],[]],
-     "nutritionQuantities" : (["food", "category"], ["qty"])}
-        pdf3 = PanDatFactory(**re_fielded_schema)
-        panDat3 = pdf3.json.create_pan_dat(filePath)
-        for t, (pks, dfs) in re_fielded_schema.items():
-            self.assertTrue(list(pks) + list(dfs) == list(getattr(panDat3, t).columns))
-
-        tdf = TicDatFactory(**netflowSchema())
-        pdf = PanDatFactory(**netflowSchema())
-        ticDat = tdf.freeze_me(tdf.TicDat(**{t:getattr(netflowData(),t) for t in tdf.primary_key_fields}))
-        panDat = pan_dat_maker(netflowSchema(), ticDat)
-        filePath = os.path.join(_scratchDir, "netflow.json")
-        pdf.json.write_file(panDat, filePath)
-        panDat2 = pdf.json.create_pan_dat(filePath)
-        self.assertTrue(pdf._same_data(panDat, panDat2, epsilon=1e-5))
-        panDat3 = pdf.json.create_pan_dat(pdf.json.write_file(panDat, ""))
-        self.assertTrue(pdf._same_data(panDat, panDat3))
-        dicted = json.loads(pdf.json.write_file(panDat, ""))
-        panDat4 = pdf.PanDat(**dicted)
-        self.assertTrue(pdf._same_data(panDat, panDat4))
-        pdf2 = PanDatFactory(**{t:'*' for t in pdf.all_tables})
-        panDat5 = pdf2.PanDat(**dicted)
-        self.assertTrue(pdf._same_data(panDat, panDat5))
+        for func in ["write_file", "write_file_pd"]:
+            tdf = TicDatFactory(**dietSchema())
+            pdf = PanDatFactory(**dietSchema())
+            ticDat = tdf.freeze_me(tdf.TicDat(**{t:getattr(dietData(),t) for t in tdf.primary_key_fields}))
+            panDat = pan_dat_maker(dietSchema(), ticDat)
+            filePath = os.path.join(_scratchDir, "diet.json")
+            getattr(pdf.json, func)(panDat, filePath)
+            panDat2 = pdf.json.create_pan_dat(filePath)
+            self.assertTrue(pdf._same_data(panDat, panDat2, epsilon=1e-5))
+            pdf2 = PanDatFactory(**{t:'*' for t in pdf.all_tables}) if func == "write_file_pd" else pdf
+            getattr(pdf.json, func)(panDat, filePath)
+            panDat2 = pdf2.json.create_pan_dat(filePath)
+            self.assertTrue(pdf._same_data(panDat, panDat2, epsilon=1e-5))
+
+            re_fielded_schema = {"categories" : (("name",),["maxNutrition", "minNutrition"]),
+         "foods" :[["name"],[]],
+         "nutritionQuantities" : (["food", "category"], ["qty"])}
+            pdf3 = PanDatFactory(**re_fielded_schema)
+            panDat3 = pdf3.json.create_pan_dat(filePath)
+            for t, (pks, dfs) in re_fielded_schema.items():
+                self.assertTrue(list(pks) + list(dfs) == list(getattr(panDat3, t).columns))
+
+            tdf = TicDatFactory(**netflowSchema())
+            pdf = PanDatFactory(**netflowSchema())
+            ticDat = tdf.freeze_me(tdf.TicDat(**{t:getattr(netflowData(),t) for t in tdf.primary_key_fields}))
+            panDat = pan_dat_maker(netflowSchema(), ticDat)
+            filePath = os.path.join(_scratchDir, "netflow.json")
+            getattr(pdf.json, func)(panDat, filePath)
+            panDat2 = pdf.json.create_pan_dat(filePath)
+            self.assertTrue(pdf._same_data(panDat, panDat2, epsilon=1e-5))
+            panDat3 = pdf.json.create_pan_dat(getattr(pdf.json, func)(panDat, ""))
+            self.assertTrue(pdf._same_data(panDat, panDat3))
+            dicted = json.loads(getattr(pdf.json, func)(panDat, ""))
+            panDat4 = pdf.PanDat(**dicted)
+            self.assertTrue(pdf._same_data(panDat, panDat4))
+            pdf2 = PanDatFactory(**{t:'*' for t in pdf.all_tables}) if func == "write_file_pd" else pdf
+            panDat5 = pdf2.PanDat(**dicted)
+            self.assertTrue(pdf._same_data(panDat, panDat5))
 
 
         tdf = TicDatFactory(**dietSchema())
@@ -595,31 +625,32 @@ def testJsonSimple(self):
         ticDat = tdf.freeze_me(tdf.TicDat(**{t:getattr(dietData(),t) for t in tdf.primary_key_fields}))
         panDat = pan_dat_maker(dietSchema(), ticDat)
         filePath = os.path.join(_scratchDir, "diet.json")
-        pdf.json.write_file(panDat, filePath, orient='columns', index=True)
+        pdf.json.write_file_pd(panDat, filePath, orient='columns', index=True)
         # the following doesn't generate a TicDatError, which is fine
         self.assertTrue(firesException(lambda : pdf.json.create_pan_dat(filePath)))
         panDat2 = pdf.json.create_pan_dat(filePath, orient='columns')
         self.assertTrue(pdf._same_data(panDat, panDat2, epsilon=1e-5))
-        panDat3 = pdf.json.create_pan_dat(pdf.json.write_file(panDat, "", orient='columns'), orient="columns")
+        panDat3 = pdf.json.create_pan_dat(pdf.json.write_file_pd(panDat, "", orient='columns'), orient="columns")
         self.assertTrue(pdf._same_data(panDat, panDat3, epsilon=1e-5))
-        dicted = json.loads(pdf.json.write_file(panDat, "", orient='columns'))
+        dicted = json.loads(pdf.json.write_file_pd(panDat, "", orient='columns'))
         panDat4 = pdf.PanDat(**dicted)
         self.assertTrue(pdf._same_data(panDat, panDat4, epsilon=1e-5))
 
     def testJsonCross(self):
         if not self.can_run:
             return
-        tdf = TicDatFactory(**dietSchema())
-        pdf = PanDatFactory(**dietSchema())
-        ticDat = tdf.freeze_me(tdf.TicDat(**{t: getattr(dietData(),t) for t in tdf.primary_key_fields}))
-        panDat = pan_dat_maker(dietSchema(), ticDat)
-        filePath = os.path.join(_scratchDir, "diet_cross.json")
-        pdf.json.write_file(panDat, filePath)
-        ticDat2 = tdf.json.create_tic_dat(filePath, from_pandas=True)
-        self.assertTrue(tdf._same_data(ticDat, ticDat2, epsilon=0.0001))
-        tdf.json.write_file(ticDat, filePath, allow_overwrite=True, to_pandas=True)
-        panDat2 = pdf.json.create_pan_dat(filePath)
-        self.assertTrue(pdf._same_data(panDat, panDat2, epsilon=0.0001))
+        for func in ["write_file", "write_file_pd"]:
+            tdf = TicDatFactory(**dietSchema())
+            pdf = PanDatFactory(**dietSchema())
+            ticDat = tdf.freeze_me(tdf.TicDat(**{t: getattr(dietData(),t) for t in tdf.primary_key_fields}))
+            panDat = pan_dat_maker(dietSchema(), ticDat)
+            filePath = os.path.join(_scratchDir, "diet_cross.json")
+            getattr(pdf.json, func)(panDat, filePath)
+            ticDat2 = tdf.json.create_tic_dat(filePath, from_pandas=True)
+            self.assertTrue(tdf._same_data(ticDat, ticDat2, epsilon=0.0001))
+            tdf.json.write_file(ticDat, filePath, allow_overwrite=True, to_pandas=True)
+            panDat2 = pdf.json.create_pan_dat(filePath)
+            self.assertTrue(pdf._same_data(panDat, panDat2, epsilon=0.0001))
 
     def testJsonSpacey(self):
         if not self.can_run:
@@ -631,10 +662,10 @@ def testJsonSpacey(self):
         panDat = pan_dat_maker(spacesSchema(), ticDat)
         ext = ".json"
         filePath = os.path.join(_scratchDir, "spaces_2%s" % ext)
-        pdf.json.write_file(panDat, filePath, case_space_table_names=True)
+        pdf.json.write_file_pd(panDat, filePath, case_space_table_names=True)
         panDat2 = pdf.json.create_pan_dat(filePath)
         self.assertTrue(pdf._same_data(panDat, panDat2))
-        panDat3 = pdf.json.create_pan_dat(pdf.json.write_file(panDat, "", case_space_table_names=True))
+        panDat3 = pdf.json.create_pan_dat(pdf.json.write_file_pd(panDat, "", case_space_table_names=True))
         self.assertTrue(pdf._same_data(panDat, panDat3))
 
 
@@ -643,13 +674,13 @@ def testJsonSpacey(self):
         ticDat = tdf.freeze_me(tdf.TicDat(**{t:getattr(netflowData(),t) for t in tdf.primary_key_fields}))
         panDat = pan_dat_maker(netflowSchema(), ticDat)
         filePath = os.path.join(_scratchDir, "spaces_2_2%s" % ext)
-        pdf.json.write_file(panDat, filePath, case_space_table_names=True)
+        pdf.json.write_file_pd(panDat, filePath, case_space_table_names=True)
         panDat2 = pdf.json.create_pan_dat(filePath)
         self.assertTrue(pdf._same_data(panDat, panDat2))
-        panDat3 = pdf.json.create_pan_dat(pdf.json.write_file(panDat, "", case_space_table_names=True))
+        panDat3 = pdf.json.create_pan_dat(pdf.json.write_file_pd(panDat, "", case_space_table_names=True))
         self.assertTrue(pdf._same_data(panDat, panDat3))
 
-        dicted = json.loads(pdf.json.write_file(panDat, "", orient='columns'))
+        dicted = json.loads(pdf.json.write_file_pd(panDat, "", orient='columns'))
         panDat4 = pdf.PanDat(**dicted)
         self.assertTrue(pdf._same_data(panDat, panDat4, epsilon=1e-5))
 
diff --git a/ticdat/testing/testpandat_utils.py b/ticdat/testing/testpandat_utils.py
index 556c1f2e..5f8edb1d 100644
--- a/ticdat/testing/testpandat_utils.py
+++ b/ticdat/testing/testpandat_utils.py
@@ -5,7 +5,7 @@
 from ticdat.testing.ticdattestutils import fail_to_debugger, flagged_as_run_alone, netflowPandasData
 from ticdat.testing.ticdattestutils import netflowSchema, copy_to_pandas_with_reset, dietSchema, netflowData
 from ticdat.testing.ticdattestutils import addNetflowForeignKeys, sillyMeSchema, dietData, pan_dat_maker
-from ticdat.testing.ticdattestutils import addDietForeignKeys
+from ticdat.testing.ticdattestutils import addDietForeignKeys, dietData
 from ticdat.ticdatfactory import TicDatFactory
 import itertools
 from math import isnan
@@ -22,6 +22,91 @@ def _deep_anonymize(x)  :
 class TestUtils(unittest.TestCase):
     canRun = False
 
+    def test_advanced_kwargs_cloning(self):
+        if not self.canRun:
+            return
+        tdf = TicDatFactory(foo=[["Primary Key Field"], ["Data Field"]])
+        tdf.set_data_type("foo", "Primary Key Field", number_allowed=True, min=0, max=float("inf"), inclusive_min=True,
+                          inclusive_max=False)
+        tdf.add_data_row_predicate("foo", lambda row: True, predicate_name="normal")
+        tdf.add_data_row_predicate("foo", lambda row, dummy: dummy, predicate_name="advanced one",
+                                   predicate_kwargs_maker=lambda dat: {"dummy": True})
+        tdf.add_data_row_predicate("foo", lambda row, min_pk: row["Primary Key Field"] != min_pk+1,
+                                   predicate_name="advanced two",
+                                   predicate_kwargs_maker=lambda dat: {"min_pk": min(dat.foo)})
+        tdf.clone()
+        dat_one = tdf.TicDat(foo=[[1, "junk"], [12, "junk"]])
+        tdf.freeze_me(dat_one)
+        dat_two = tdf.TicDat(foo=[[1, "junk"], [2, "junk"]])
+        tdf.freeze_me(dat_two)
+        self.assertFalse(tdf.find_data_row_failures(dat_one, exception_handling="Handled as Failure"))
+        self.assertTrue(tdf.find_data_row_failures(dat_two, exception_handling="Handled as Failure"))
+        pdat_one = tdf.copy_to_pandas(dat_one, reset_index=True)
+        pdat_two = tdf.copy_to_pandas(dat_two, reset_index=True)
+        # the advanced row predicates don't copy over when cloning to an explicitly different type
+        pdf = tdf.clone(clone_factory=PanDatFactory)
+        self.assertFalse(pdf.find_data_row_failures(pdat_one, exception_handling="Handled as Failure"))
+        self.assertFalse(pdf.find_data_row_failures(pdat_two, exception_handling="Handled as Failure"))
+        pdf.add_data_row_predicate("foo", lambda row, dummy: dummy, predicate_name="advanced one",
+                                   predicate_kwargs_maker=lambda dat: {"dummy": True})
+        pdf.add_data_row_predicate("foo", lambda row, min_pk: row["Primary Key Field"] != min_pk+1,
+                                   predicate_name="advanced two",
+                                   predicate_kwargs_maker=lambda dat: {"min_pk": min(dat.foo["Primary Key Field"])})
+        pdf = pdf.clone()
+        self.assertFalse(pdf.find_data_row_failures(pdat_one, exception_handling="Handled as Failure"))
+        self.assertTrue(pdf.find_data_row_failures(pdat_two, exception_handling="Handled as Failure"))
+
+        pdf_w_bum_preds = tdf.clone(clone_factory=PanDatFactory.create_from_full_schema)
+        self.assertFalse(any(pdf_w_bum_preds.find_data_type_failures(_) for _ in [pdat_one, pdat_two]))
+        self.assertTrue(all(pdf_w_bum_preds.find_data_row_failures(_, exception_handling="Handled as Failure")
+                            for _ in [pdat_one, pdat_two]))
+        tdf2 = pdf.clone(clone_factory=TicDatFactory)
+        # the advanced row predicates don't copy over when cloning to an explicitly different type
+        self.assertFalse(tdf2.find_data_row_failures(dat_one, exception_handling="Handled as Failure"))
+        self.assertFalse(tdf2.find_data_row_failures(dat_two, exception_handling="Handled as Failure"))
+        tdf_w_bum_preds = pdf.clone(clone_factory=TicDatFactory.create_from_full_schema)
+        self.assertFalse(any(tdf_w_bum_preds.find_data_type_failures(_) for _ in [dat_one, dat_two]))
+        self.assertTrue(all(tdf_w_bum_preds.find_data_row_failures(_, exception_handling="Handled as Failure")
+                            for _ in [dat_one, dat_two]))
+
+
+    def testDefaultAdd(self):
+        if not self.canRun:
+            return
+        tdf = TicDatFactory(**dietSchema())
+        ticDat = tdf.freeze_me(tdf.TicDat(**{t:getattr(dietData(),t) for t in tdf.primary_key_fields}))
+        panDat = pan_dat_maker(dietSchema(), ticDat)
+        tdf2 = TicDatFactory(**{k:[p,d] if k!="foods" else [p, list(d)+["extra"]] for k,(p,d) in dietSchema().items()})
+        pdf2 = tdf2.clone(clone_factory=PanDatFactory)
+        panDat2 = pdf2.PanDat(**{t: getattr(panDat, t) for t in tdf.all_tables})
+        ticDat2 = tdf2.TicDat(**{t: getattr(ticDat, t) for t in tdf.all_tables})
+        self.assertTrue(tdf2._same_data(ticDat2, pdf2.copy_to_tic_dat(panDat2), epsilon=1e-5))
+        self.assertTrue(set(panDat2.foods["extra"]) == {0})
+        pdf3 = pdf2.clone()
+        pdf3.set_default_value("foods", "extra", 100)
+        panDat3 = pdf3.PanDat(**{t: getattr(panDat, t) for t in tdf.all_tables})
+        self.assertFalse(tdf2._same_data(ticDat2, pdf3.copy_to_tic_dat(panDat3), epsilon=1e-5))
+        self.assertTrue(set(panDat3.foods["extra"]) == {100})
+
+    def testCopying(self):
+        tdf = TicDatFactory(**dietSchema())
+        pdf = PanDatFactory(**dietSchema())
+        dat = tdf.copy_tic_dat(dietData())
+        def make_pan_dat(reset_index):
+            rtn = tdf.copy_to_pandas(dat, reset_index=reset_index)
+            return pdf.PanDat(**{k: getattr(rtn, k) for k in tdf.all_tables})
+        dat2 = pdf.copy_to_tic_dat(make_pan_dat(False))
+        dat3 = pdf.copy_to_tic_dat(tdf.copy_to_pandas(dat, reset_index=True))
+        self.assertTrue(tdf._same_data(dat, dat2))
+        self.assertTrue(tdf._same_data(dat, dat3))
+        def make_tic_dat_dat(reset_index):
+            rtn = tdf.copy_to_pandas(dat, reset_index=reset_index)
+            return tdf.TicDat(**{k: getattr(rtn, k) for k in tdf.all_tables})
+        dat2 = make_tic_dat_dat(False)
+        dat3 = make_tic_dat_dat(True)
+        self.assertTrue(tdf._same_data(dat, dat2))
+        self.assertTrue(tdf._same_data(dat, dat3))
+
     def testSimple(self):
         if not self.canRun:
             return
diff --git a/ticdat/testing/testutils.py b/ticdat/testing/testutils.py
index 43d62871..a9fdfbe9 100644
--- a/ticdat/testing/testutils.py
+++ b/ticdat/testing/testutils.py
@@ -432,6 +432,7 @@ def _getfk_cards(t):
         self.assertFalse(tdf.foreign_keys)
 
     def testFiveB(self):
+        # this tests obfusimplify and add_implied_foreign_keys both
         tdf = TicDatFactory(parent = [["Boo"],["Field"]], other_parent = [["Bar"],["Field"]],
                             child = [["Boo", "Bar"], ["Field"]], more_child = [["Bar", "Boo"], ["Field"]])
         tdf.add_foreign_key("child", "parent", ["Boo"]*2)
diff --git a/ticdat/testing/testxls.py b/ticdat/testing/testxls.py
index a7233509..0718086c 100644
--- a/ticdat/testing/testxls.py
+++ b/ticdat/testing/testxls.py
@@ -483,15 +483,14 @@ def round_trip():
         self.assertTrue(dat.parameters.shape == (4, 2))
 
     def test_parameters(self):
-        filePath = os.path.join(_scratchDir, "parameters.xls")
+        filePath = os.path.join(_scratchDir, "parameters.xlsx")
         tdf = TicDatFactory(parameters=[["Key"], ["Value"]])
         tdf.add_parameter("Something", 100)
         tdf.add_parameter("Different", 'boo', strings_allowed='*', number_allowed=False)
         dat = tdf.TicDat(parameters = [["Something",float("inf")], ["Different", "inf"]])
-        for _ in ["", "x"]:
-            tdf.xls.write_file(dat, filePath+_)
-            dat_ = tdf.xls.create_tic_dat(filePath+_)
-            self.assertTrue(tdf._same_data(dat, dat_))
+        tdf.xls.write_file(dat, filePath)
+        dat_ = tdf.xls.create_tic_dat(filePath)
+        self.assertTrue(tdf._same_data(dat, dat_))
 
     def testDietWithInfFlagging(self):
         tdf = TicDatFactory(**dietSchema())