propelorm/Propel2 — Issue #52: Extend connections management
Author: RafalFilipek
Created: 2011-11-16 23:04:10+00:00
State: open

I want to share some ideas about using ORM's in big projects. I know Propel has basic support for "master" "multiple slaves". But in big projects it would be grate to have more control. I still do not have any briliant idea, how to implement it, so I'll start from tasks that are importat form my point of view.

1. There shoudl be easy way to define master and slaves connections ( ok, we can do it now )
* master can be READ-WRITE on WRITE-only
* slaves are READ only
2. Why we should be able to define master in READ-WRITE mode ?
* In may cases master is located on more powerfull machine. So it can handle some part of READ queries.
* Sometimes you have important informations that you must access just after you put them into database. In that case you cant trust that slave will get up-to-date informations.
3. According to point 2 there should be some way to define probability of selecting master in case of READ query.
4. There should be some way to force Propel to user master for upcoming READ query.
5. There should be some way to handle connection problems. For example, if we have master and 2 slaves:
* if there is WRITE query and master is not working - we're dead no way to handle it.
* if there is READ query and selected db is not connected another one should be selected. Also dead connection shoud be placed on 'black-list'. This point can be in fact quite complex in case we would like to keep black-list for some time ( for example 20 seconds ), because there should be some cache system involved.
* if there is READ query and all databases are not woring - we're dead.

Some cases ( Asume that probability of selecting master for READ query is 30% and we have 2 slaves):

1. Simple example
* 1st query ( READ ) - slave_1 is selected
* 2nd query ( READ ) - slave_1 ( we're using same connection as long as we can )
* 3rd query ( WRITE ) - master is selected
* 4th query ( READ ) - lets back to slave_1

2. Force master selection ( in case master is READ-WRITE )
* 1st query ( READ ) - slave_2 is selected
* 2nd query ( WRITE ) - master is selected
* 3th query ( READ but we're asking about some VERY IMPORTANT DATA) - use_master=true master is selected
* 4th query ( READ ) -lest back to slave_2

3. Master also can be picked for READ queries
* 1st query ( READ ) - wow rand < 30 so master is selected
* 2nd query ( READ ) still master ( we're keep same connection just like with slave connection )
* 3rd, 4th ... ( some READ or WRITE ) we are still usign master connection

In my opinion those points CAN'T be placed directly inside Propel, Connection or any other "core" class. What I was thinking is something like "````ConnectionManager````" class. Maybe this name is not the best idea but developers could keep all those logic and functionality in this class.

Those points are not my "I want it or I'll get angry". In our company we're working on applications that must handle objectively high traffic. We know about all cache stuff and so on. Last project is using Doctrine2 and it was pain because of lack of expansibility. Propel has many grate ideas and in my opinion this is the best moment to think about that. I know Propel can handle small and normal traffic, but 'hey' big projects also want to use it :)