wikimedia/puppet-cdh4 — pull request #15 by supertom (2013-03-29 20:32:59+00:00)
Title: partial template support

Hi Andrew,

Another feature I needed was the ability to include additional properties in the files that are currently templated. One way to do that is to simply expose every option in those files, but that makes the argument list of the classes very long, as well as leads to constant code changes. Instead, I accomplished this by using the "partial templates" concept here:

http://somethingsinistral.net/blog/partial-templates-with-puppet/

In short, a user drops in a file with the content they want and it gets included in the template.

### Usage

In terms of usage, a user of this feature would do the following:
1. in the templates directory, add your additions into a file called "_local_FILENAME". For example, to have content injected into the hdfs-site.xml file, you would put your content in a file called "_local_hdfs-site.xml.erb" in the templates directory.

If the file is present, it will be included. It itself is a template, so template syntax should work inside that included template. If the file is not present (or is removed later), the template file will be updated on the machine running the puppet client to reflect that. I have some example output below to demonstrate this.

### Implementation

In terms of implementation, here is how it works:

* Each XML file now includes the following line (just above the closing "configuration" tag):

```ruby
<%= scope.function_include_local_template([__FILE__]) %>
```

* There is a **custom function** at **lib/puppet/parser/functions/include_local_template.rb** that does the work of determining if the local file is present and slurping it into the XML template.

Possible improvements

We might want to make the filename and path more configurable. Right now we derive the name and assume it is in the same directory. Making this change would allow someone to further customize the .ini files (multiple includes), as well as locate the "_local_" (partial template) files somewhere else. If you think that feature is useful I'm open to adding that functionality.

Please let me know if you have any questions.

Thanks,

Tom

Output Example:
```
#######
example output when adding, and then removing the local template
#######

supertom@compute1:~$ sudo puppet agent --test --no-usecacheonfailure
info: Retrieving plugin
info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
info: Caching catalog for compute1.cluster.foo
info: Applying configuration version '1364577789'
notice: /Stage[main]/supertom-common/Exec[supertom-members]/returns: executed successfully
notice: /Stage[main]/Cdh4::Hadoop::Config/File[/etc/hadoop/conf/mapred-site.xml]/content:
--- /etc/hadoop/conf/mapred-site.xml 2013-03-29 16:56:10.352352021 +0000
+++ /tmp/puppet-file20130329-8429-1g3rne2-0 2013-03-29 17:35:02.432706451 +0000
@@ -37,6 +37,42 @@
This is useful for decommissioning nodes.

-
+
+ mapred.map.tasks.speculative.execution
+ false
+
+
+ mapred.reduce.tasks.speculative.execution
+ false
+
+
+ mapred.jobtracker.taskScheduler
+ org.apache.hadoop.mapred.FairScheduler
+
+
+ mapred.fairscheduler.weight.adjuster
+ org.apache.hadoop.mapred.NewJobWeightBooster
+
+
+ mapred.fairscheduler.allocation.file
+ /etc/hadoop/conf/fair-scheduler.xml
+
+
+ mapred.fairscheduler.assignmultiple
+ true
+
+
+ mapred.fairscheduler.preemption
+ true
+
+
+ mapred.child.env
+ LD_LIBRARY_PATH=.:/usr/lib/jni/:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/server/
+ env vars needed by the mapper or reducer
+
+
+ mapred.task.timeout
+ 90000000
+

info: FileBucket got a duplicate file {md5}d3a673bfa762af18fed3277335d930b0
info: /Stage[main]/Cdh4::Hadoop::Config/File[/etc/hadoop/conf/mapred-site.xml]: Filebucketed /etc/hadoop/conf/mapred-site.xml to puppet with sum d3a673bfa762af18fed3277335d930b0
notice: /Stage[main]/Cdh4::Hadoop::Config/File[/etc/hadoop/conf/mapred-site.xml]/content: content changed '{md5}d3a673bfa762af18fed3277335d930b0' to '{md5}2aa6197405c4d124340c593a62c25557'
notice: Finished catalog run in 28.35 seconds
supertom@compute1:~$ sudo puppet agent --test --no-usecacheonfailure
info: Retrieving plugin
info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
info: Caching catalog for compute1.cluster.foo
info: Applying configuration version '1364578595'
notice: /Stage[main]/supertom-common/Exec[supertom-members]/returns: executed successfully
notice: /Stage[main]/Cdh4::Hadoop::Config/File[/etc/hadoop/conf/mapred-site.xml]/content:
--- /etc/hadoop/conf/mapred-site.xml 2013-03-29 17:35:02.500707166 +0000
+++ /tmp/puppet-file20130329-9260-1ge5s63-0 2013-03-29 17:36:56.793911362 +0000
@@ -37,42 +37,6 @@
This is useful for decommissioning nodes.

-
- mapred.map.tasks.speculative.execution
- false
-
-
- mapred.reduce.tasks.speculative.execution
- false
-
-
- mapred.jobtracker.taskScheduler
- org.apache.hadoop.mapred.FairScheduler
-
-
- mapred.fairscheduler.weight.adjuster
- org.apache.hadoop.mapred.NewJobWeightBooster
-
-
- mapred.fairscheduler.allocation.file
- /etc/hadoop/conf/fair-scheduler.xml
-
-
- mapred.fairscheduler.assignmultiple
- true
-
-
- mapred.fairscheduler.preemption
- true
-
-
- mapred.child.env
- LD_LIBRARY_PATH=.:/usr/lib/jni/:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/:/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/server/
- env vars needed by the mapper or reducer
-
-
- mapred.task.timeout
- 90000000
-
+

info: FileBucket got a duplicate file {md5}2aa6197405c4d124340c593a62c25557
info: /Stage[main]/Cdh4::Hadoop::Config/File[/etc/hadoop/conf/mapred-site.xml]: Filebucketed /etc/hadoop/conf/mapred-site.xml to puppet with sum 2aa6197405c4d124340c593a62c25557
notice: /Stage[main]/Cdh4::Hadoop::Config/File[/etc/hadoop/conf/mapred-site.xml]/content: content changed '{md5}2aa6197405c4d124340c593a62c25557' to '{md5}d3a673bfa762af18fed3277335d930b0'
notice: Finished catalog run in 27.65 seconds
```