Append to application.conf instead of blowing away