ini_set only sets a value for a php ini configuration key so I think it should not be dangerous. In any case, I could check whether 'memory_limit' option is null before call ini_set function. In this manner, ini_set only would be setted if memory_limit option would not be null. 

php app/console -m 1 -a 128 -r routingkey .... --> This call would call ini_set('memory_limit', '128M')
php app/console -m 1 -r routingkey .... --> This call would not set memory limit

What do you think?