ENH: Add support for defaultdict as "dtype" in parser
State: closed

- [x] closes #41574 
- [x] tests added / passed (1 test is modified)
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

# What is new:

Now defaultdict can be provides as dtype.

The implementation allows moving to supporting Iterable as dtype



Labels:
name: Enhancement
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO CSV
description: read_csv, to_csv
name: Stale

Comments:
> 
> 
> Thanks for the pr @NickVeld! Looks like test you added is failing. Also some quick comments, plus if you could add a whatsnew note in the enhancement section mentioning this improvement.

Fixed and done.

But I suppose that I also should find all usages of TextReader and put `.. versionadded::1.3.0` with defaultdict support everywhere
Right now looks like there is distinct handling in 3 different places. Can we factor out this logic to one function responsible for sanitizing the `dtypes` argument? Then we could use it in any places `dtypes` is accepted for consistency (like read_json, read_excel)
If you are speaking about my changes in `parser.pyx`, `c_parser_wrapper.py` and `python_parser.py` ...

---------------

`ensure_dtype_objs` from `c_parser_wrapper.py` can be moved into `TextReader`

![image](https://user-images.githubusercontent.com/17028401/119771892-0c5b2000-bec7-11eb-8964-aecf97726fac.png)

But the constructor of `TextReader` [states the following](https://github.com/pandas-dev/pandas/blob/ad8a4ea41583463c2e10cc7882a20e9f8cbe4434/pandas/_libs/parsers.pyx#L490):
```
# Caller is responsible for ensuring we have one of
# - None
# - DtypeObj
# - dict[Any, DtypeObj]
```

Why it has been implemented in this way?

---------------

But I am not sure about the changes in `python_parser.py` because it seems that `PythonParser` does not use anything used by `CParser` or `TextReader`. So everything must be before the parser usage. Are you sure it is good to place the sanitization and the usage that far?
> Why it has been implemented in this way?

Not too familiar with parser code, so not sure. My thought was just that from a maintenance perspective, we'd like to be adding logic for handling this case in one location. And ideally that could be used for any function accepting `dtypes` as a dict such that the API can be as consistent as possible. So based on that my thought was adding a single common function responsible for sanitizing (and validating) this argument at the top level. IMO since the workaround from a user perspective can be done in a straightforward way, if we have to add a bunch of separate logic in different places to handle `defaultdict`, it's not worth it for the extra maintenance burden.


@mzeitlin11 
Sorry, for the delay.

> My thought was just that from a maintenance perspective, we'd like to be adding logic for handling this case in one location

As I said I believe I can merge two occurrences (of three) into the one. But not the third one. Because the project is not designed to allow it easily. Moreover the third one was found by a mechanical search and I assume it can be used somewhere else.

> IMO since the workaround from a user perspective can be done in a straightforward way

What workaround? The authors fails to create such. Me too, it was likely that it was not a blocker for me.  

> a bunch of separate logic in different places to handle defaultdict, it's not worth it for the extra maintenance burden.

Much more the maintenance burden comes from the merging

------------

Thus, please, could you dive into code and propose the solution of merging three places into one. Or lets agree on the merging only two of them.

@NickVeld Can you rebase and try to handle this in ``base_parser.py`` which the CParserWrapper and PythonParser inherit from? Thanks.
> @NickVeld Can you rebase and try to handle this in `base_parser.py` which the CParserWrapper and PythonParser inherit from? Thanks.

Rebase to updated `master`?

I can move all "cleaners" from `CParserWrapper` and `PythonParser` to `PythonBase`.
(However, it can demand changes in the tests if the tests needs moving too,)
But the value extraction logic from `parser.pyx` needs changing. So the changes to `master` will be in `parser.pyx` and `base_parser.py`.
Yes, you need to pull in the latest master since there are conflicts. Typically, resolving the conflicts in the Github Web UI will do it for you, but you will also need to pull the latest master in occasionally manually even if there are no conflicts.
About combining the logic, I think I'm fine with changes in ``parser.pyx`` and ``base_parser.py``, since thats probably the most everything can be combined.
Looking at the tests, though, it looks like this is only tested for the C parser and not the python parser. Can you move the tests so that both parsers are tested?
Thanks.
Ok
Must `converters` type verification be moved from [here](https://github.com/pandas-dev/pandas/blob/a6a8915ca06639c4b29c35b839ce404cfeae6afc/pandas/io/parsers/readers.py#L989) to `base_parser.py/ParserBase` too?

Pros:
* All processing in one place

Cons:
* The stack of raised `TypeError` will be longer
@lithomas1 

> Looking at the tests, though, it looks like this is only tested for the C parser and not the python parser. Can you move the tests so that both parsers are tested?

According to the tests all parsers (through `all_parsers` fixture) are tested so there is no need in the move.
Hello @NickVeld! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-06-17 05:04:54 UTC
@NickVeld Can you please move the convertors change out of this PR as @jreback mentioned. It requires a deprecation cycle and shouldn't be mixed with these changes.
Also tests are failing in the CI, so can you fix that?
Thanks.

> pls just a single change in the PR. i am not really sure what you are trying to change.
> Can you please move the convertors change out of this PR as @jreback mentioned

I have a state without this change more than a week. You (as organization) has blocked the PR waiting this change and now asks to remove it. Why?

> It requires a deprecation cycle and shouldn't be mixed with these changes.

Now there is no deprecation. You can merge the current version (after the fixing) and merge the deprecation of the convertors in another PR

> Also tests are failing in the CI, so can you fix that?

Of course
There is a problem:
I took from `python_parser.py` a code that transforms dict keys from column indices to column names but it relies on `self.orig_names` that is defined differently in each the parser.
What definition of `self.orig_names` should be used?
@lithomas1 Hey?
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
> 
> 
> This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

Hello?
this is quite old, happen to reopen if actively worked on.