ansible/ansible — pull request #8364 by piyush0101 (2014-07-30 19:35:22+00:00)
Title: Logging of module invocation for huge datasets to ansible module is very slow

TL;DR:
Ansible is inefficient in logging the invocation of a module if input data to the module is pretty huge. This is because of a very general regex expression.

Full Story:

I was collecting some data from several thousand hosts and passing it on to an AnsibleModule. Module was very slow. I did some profiling with cProfile and pstats and found that invocation of __log_invocation() method call in the class AnsibleModule is taking most of the time. It is doing expensive regular expression matching trying to obfuscate sensitive data like passwords from ansible syslogs. For complex input data which in this case is a dictionary of all hosts for a region it does not scale. The problem is the regular expression which is very general and open which results in a lot of potential matches and for a large dataset it becomes very slow. Here's the output from the profiler.

update_regions.prof% sort cumulative
update_regions.prof% stats 10
Tue Jul 29 17:50:26 2014 update_regions.prof

54670 function calls (53455 primitive calls) in 215.633 seconds
Ordered by: cumulative time
List reduced from 959 to 10 due to restriction

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.002 0.002 215.633 215.633 update_looksee_dfw:32()
1 0.000 0.000 215.579 215.579 update_looksee_dfw:67(main)
1 0.000 0.000 213.086 213.086 update_looksee_dfw:254(__init__)
1 0.045 0.045 213.055 213.055 update_looksee_dfw:848(_log_invocation)
131 212.851 1.625 212.851 1.625 {method 'match' of '_sre.SRE_Pattern' objects}