mesos/hadoop — pull request #14 by brndnmtthws (2014-01-17 18:53:15+00:00)
Title: Fixed port overlap, tracker GC

@florianleibert @benh @bmahler @airbnb/di

There's an edge case present where the TT ports are in in use, the TT fails to start, and the JT starts to implode. This is because we don't handle the case where a TT is started on the same/host port after another one was started and the periodic timer starts. To further prevent problems, I've randomized the port selection from the offered port range.

Here's the TT log:

```
14/01/17 18:32:04 WARN conf.Configuration: dfs.df.interval is deprecated. Instead, use fs.df.interval
14/01/17 18:32:04 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
14/01/17 18:32:04 WARN conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
14/01/17 18:32:04 WARN conf.Configuration: topology.script.file.name is deprecated. Instead, use net.topology.script.file.name
14/01/17 18:32:04 WARN conf.Configuration: topology.script.number.args is deprecated. Instead, use net.topology.script.number.args
14/01/17 18:32:04 WARN conf.Configuration: dfs.umaskmode is deprecated. Instead, use fs.permissions.umask-mode
14/01/17 18:32:04 WARN conf.Configuration: topology.node.switch.mapping.impl is deprecated. Instead, use net.topology.node.switch.mapping.impl
14/01/17 18:32:05 WARN conf.Configuration: slave.host.name is deprecated. Instead, use dfs.datanode.hostname
14/01/17 18:32:05 WARN conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id
14/01/17 18:32:05 FATAL mapred.MesosExecutor: Failed to start TaskTracker
java.net.BindException: Problem binding to [i-8977e1f3:31001] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:715)
at org.apache.hadoop.ipc.Server.bind(Server.java:356)
at org.apache.hadoop.ipc.Server$Listener.(Server.java:454)
at org.apache.hadoop.ipc.Server.(Server.java:1835)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:970)
at org.apache.hadoop.ipc.WritableRpcEngine$Server.(WritableRpcEngine.java:365)
at org.apache.hadoop.ipc.WritableRpcEngine.getServer(WritableRpcEngine.java:287)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:695)
at org.apache.hadoop.ipc.RPC.getServer(RPC.java:684)
at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:971)
at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1720)
at org.apache.hadoop.mapred.MesosExecutor.launchTask(MesosExecutor.java:81)
```

And here's some metrics showing what occurs in the JT:

![render 3](https://f.cloud.github.com/assets/3129093/1943318/9c2824dc-7fa8-11e3-8e95-ee6e836e9d05.png)
![render 4](https://f.cloud.github.com/assets/3129093/1943319/9e138c0a-7fa8-11e3-996b-de742966dbea.png)