Issue: Peewee should not import all DB drivers unconditionally
Author: Necoro
State: closed
Created: 2016-08-18 20:34:55

Peewee currently imports (or tries to import) all database drivers unconditionally. As a result, you load and link them even when not needed (if you happen to have them installed for any reason). When using some driver from the playground you even get one more.

I'd suggest to only load them when needed (for instance as part of initializing the Database objects).

If this is not turned down for conceptual reasons, I'd be happy to implement this and open a pull request :)


> When using some driver from the playground you even get one more.

What do you mean?

I hear your concerns, but I'm not convinced the fix is necessary.


For instance, when using apsw or sqlcipher, you have the apsw/sqlcipher driver loaded additonally to the (if you are unlucky) 3 others, even though you don't need them at all.

Though the case even gets more complicated here, as apsw and sqlcipher both explicitly pull in sqlite_ext, which might take more work to get 'clean' and thus is not part of my original request.

My concern stem from the fact, that I intend to write a CLI application using peewee: loading completely unneeded modules just increases startup time, which is a nuisance for CLIs :)


Can you provide a measure of the amount of time saved by not loading, since that's your concern?


Ok, you win :). I must admit, I hadn't timed it before (it was more of a 'heh - this is superfluous' moment):
python -c 'from peewee import *'
un-scientifically times to
- 0.1s with sqlite only
- 0.16s with pymysql
- 0.16s with pymysql + psycopg2

But the times have to be taken with a grain of salt, as the variance is large (from 0.09 to 0.15s for plain sqlite only). Also, so it turns out, using python2 actually halves the times... well...

So, you may close this issue.


Believe me, this project is called peewee because I want it to be as small and simple as possible while being useful. When it comes to pythons import system, module caching, dynamic linking, etc, I think I see that as a boundary beyond which python is probably going to fight attempts to optimize. Part of the issue, too, has to do with peewee being a single module. If there were specific modules for the supported backends this wouldn't be an issue, because you'd just import peewees sqlite backends, etc, but I also made a decision to try and keep peewee as a single module.


I have a desktop app that often dynamically reloads data with each change in the search string, a live search. Using python -m pyinstrument …, I notice that peewee takes .105 seconds to load. 0.032 s. for psycopg2 (which I use) and .029 s. for pymysql, That's about 1/3 of loading time taken up by an unneeded driver. Moving it to the beginning of the MySQLDatabase class should cause it to load only when the class is instantiated, yes? For my case, any reduction of total execution time is noticable.

As I wrote earlier:

> When it comes to pythons import system, module caching, dynamic linking, etc, I think I see that as a boundary beyond which python is probably going to fight attempts to optimize.

Some projects import a ton of different modules. Look at Django...how many modules do you think Django imports? Or SQLAlchemy? Peewee is already *very lean*.

I understand peewee is already lean. It's not unprecedented to place imports within classes so they are performed only when a class is instantiated. Yes Django and SQLAlchemy are even heavier. When I started using Peewee years ago, I chose it because it is much lighter than other ORMs and offers great ease of use without being too complex. Many ORMs are often used for web apps where module load time is less important. In cases of desktop apps or scripts, when one wants to be as lightweight as possible, the loading of unneeded modules is noticable. Please consider it. For now I have modified the copy of Peewee I use and have noticed a dramatic difference in cases when I need to update a desktop UI and total execution time of less than half a second becomes noticible.

Why don't you just lazily import `peewee` in your desktop app?

My app constantly uses peewee. It's that the import loads modules for SQLite, mysql, and in my case postgresql. Each one comprises a significant amount of total import time. Sometimes I might perform multiple DB operations in one execution but most of the time, code is executed from the beginning and reloading all modules multiple times per second.

I don't know why you are paying for the cost of loading Peewee more than once. Or why it would matter outside of the first time it is imported. 

It's not that Peewee is loading more than once, it's that the entire script is running multiple times per second and thus Peewee is loading as such. My use is for an extension or plugin for a desktop search app. Unlike a true desktop app that might have an event loop, the entire script runs each time with user input, thus with each input, reducing any part of the execution time is greatly helpful. I'm not familiar with Python GUI frameworks, but perhaps in those uses, such a change might be helpful as well.

So you're shelling out to a Python script on every keypress by a user. This Python script loads up your code, establishes a database connection, and runs some queries. And you have decided to waste both of our time talking about two or three import statements.