allegro/ralph — pull request #457 by vi4m (2013-02-25 11:23:14+00:00)
Title: Puppet agents reporting should handle duplicates well.

tl;dr - We can skip puppet reports duplicates thus lowering for about ~ 10 times number of cichange records.

I observed, that number of puppet agents send puppet reports for the same configuration version and host. (in the interval of 30 mins)

Further, i did some research for big data:

mysql> select count(*),sum(x) from (select count(configuration_version) as x from cmdb_cichangepuppet where host='t10720.te2' group by configuration_version having count(*)>1) temp;
+----------+--------+
| count(*) | sum(x) |
+----------+--------+
| 57 | 3702 |
+----------+--------+
1 row in set (2.13 sec)

mysql> select sum(x) from (select count(configuration_version) as x from cmdb_cichangepuppet where host='t10720.te2' group by configuration_version having count(*)=1) temp;
+--------+
| sum(x) |
+--------+
| 145 |
+--------+
1 row in set (2.15 sec)

As observed, instead of 3847 records, we can lower this number to 202 - reducing 20x times - by storing only unique(configuration-version,host) records.