opened
gavanderhoorn
username_0
Unfinished, but wanted to get some early feedback.

Docs have not been updated.
WIP: Add pre 2014 support -- second attempt
issue
created
gavanderhoorn
username_0
@wasowski @username_2 @username_1

To use:

 - build Docker image (use build script)
 - check help output of `rosinstall_generator_tm.sh`

Note: this new version of the time machine does not accept URLs to issues/PRs as input any more. It only accepts iso8601 timestamps.

The `get_issue_creation_date.py` script can be used to find the `created_at` datetime stamp, but that requires `PyGitHub`. That is present in the Docker image, but I don't provide an easy way to run that script inside a container yet.
comment
created
gavanderhoorn
username_0
Related: #1, #2, #3 and #4.
comment
created
gavanderhoorn
username_0
Next goal would be to add the time machine script itself to the Docker image. That way it would be completely self-contained.
comment
created
gavanderhoorn
username_0
A simple test -- to see if at least the infrastructure of the new version works on machines-not-my-own:

```
rosinstall_generator_tm.sh "2016-01-01T00:00:00+00:00" indigo roscpp --deps
```

That should get you a `.rosinstall` file (but printed to your console) with `roscpp` itself *and* all of its (transitive) dependencies.

I've been creating some more rosdistro caches and uploading them so ideally generating that `.rosinstall` file takes very little time, as it should be able to reuse a cache I've already created.
comment
created
git-afsantos
username_1
The new changes look good, but once again my inexperience with Docker strikes.

From what I understood, `virtualenv` has been replaced with a Docker image, which contains all the dependencies of the time machine. Including the time machine within said image is the end goal.
So, how does this work in practice for the newbies? Do I build the image and run a container, or does the time machine interact with the image in a more transparent way?

Sorry if the help output of the script answers this, have not tried it yet, but will soon.
comment
created
gavanderhoorn
username_0
For the moment you only interact with the script. It does everything for you.

Except building the Docker image it needs that will contain the necessary `virtualenv`s.

So as I wrote in https://github.com/rosin-project/rosinstall_generator_time_machine/pull/8#issuecomment-437100402, you run `build.sh` in the `docker` sub dir and you're set (`rosinstall_generator_tm.sh` will also complain if it can't find the Docker image it needs).

---

For the next version I'll add `rosinstall_generator_tm.sh` to the image and add a very thin shell script that will basically do something like this:

```
docker run \
  -it \
  --rm \
  -v <map some dirs in your homedir to inside the container> \
  robust-rosin/rosinstall_generator_time_machine:02 \
  /rgtm/rosinstall_generator_tm.sh "$*"
```

So you still only interact with a bash script, but that just forwards everything to the script inside the Docker image.
comment
created
ChrisTimperley
username_2
This is awesome. Thanks! I'll update the `build-rosinstall.py` script on a separate branch.
comment
created
ChrisTimperley
username_2
One observation: It looks like git versions are now being returned instead of archives? Using the archives might help to reduce the size of the Docker image for each scenario?
comment
created
git-afsantos
username_1
Need to clone rosdistro .. done
Need to retrieve some rosdistro caches ..done
Requested timepoint: '2016-01-01T00:00:00+00:00' (1451606400)
Resetting local rosdistro clone ..
Already on 'master'
Your branch is up-to-date with 'origin/master'.
Determined rosdistro commit: a772634f (authored: 1451597278)
Reverting to ros/rosdistro@a772634f
Cache already exists for (distro; stamp) tuple, skipping generation
Updating local rosdistro index.yaml to use cache from the past ..
Invoking: rosinstall_generator --rosdistro=indigo roscpp --deps --tar
Using ROS_DISTRO: indigo
- tar:
    local-name: catkin
    uri: https://github.com/ros-gbp/catkin-release/archive/release/indigo/catkin/0.6.16-0.tar.gz
    version: catkin-release-release-indigo-catkin-0.6.16-0
- tar:
    local-name: cmake_modules
    uri: https://github.com/ros-gbp/cmake_modules-release/archive/release/indigo/cmake_modules/0.3.3-0.tar.gz
    version: cmake_modules-release-release-indigo-cmake_modules-0.3.3-0
- tar:
    local-name: gencpp
    uri: https://github.com/ros-gbp/gencpp-release/archive/release/indigo/gencpp/0.5.3-0.tar.gz
    version: gencpp-release-release-indigo-gencpp-0.5.3-0
- tar:
    local-name: genlisp
    uri: https://github.com/ros-gbp/genlisp-release/archive/release/indigo/genlisp/0.4.15-0.tar.gz
    version: genlisp-release-release-indigo-genlisp-0.4.15-0
- tar:
    local-name: genmsg
    uri: https://github.com/ros-gbp/genmsg-release/archive/release/indigo/genmsg/0.5.6-0.tar.gz
    version: genmsg-release-release-indigo-genmsg-0.5.6-0
- tar:
    local-name: genpy
    uri: https://github.com/ros-gbp/genpy-release/archive/release/indigo/genpy/0.5.7-0.tar.gz
    version: genpy-release-release-indigo-genpy-0.5.7-0
- tar:
    local-name: message_generation
    uri: https://github.com/ros-gbp/message_generation-release/archive/release/indigo/message_generation/0.2.10-0.tar.gz
    version: message_generation-release-release-indigo-message_generation-0.2.10-0
- tar:
    local-name: message_runtime
    uri: https://github.com/ros-gbp/message_runtime-release/archive/release/indigo/message_runtime/0.4.12-0.tar.gz
    version: message_runtime-release-release-indigo-message_runtime-0.4.12-0
- tar:
    local-name: ros/rosbuild
    uri: https://github.com/ros-gbp/ros-release/archive/release/indigo/rosbuild/1.11.11-0.tar.gz
    version: ros-release-release-indigo-rosbuild-1.11.11-0
- tar:
    local-name: ros/roslang
    uri: https://github.com/ros-gbp/ros-release/archive/release/indigo/roslang/1.11.11-0.tar.gz
    version: ros-release-release-indigo-roslang-1.11.11-0
- tar:
    local-name: ros/roslib
    uri: https://github.com/ros-gbp/ros-release/archive/release/indigo/roslib/1.11.11-0.tar.gz
    version: ros-release-release-indigo-roslib-1.11.11-0
- tar:
    local-name: ros/rosunit
    uri: https://github.com/ros-gbp/ros-release/archive/release/indigo/rosunit/1.11.11-0.tar.gz
    version: ros-release-release-indigo-rosunit-1.11.11-0
- tar:
    local-name: ros_comm/rosconsole
    uri: https://github.com/ros-gbp/ros_comm-release/archive/release/indigo/rosconsole/1.11.16-0.tar.gz
    version: ros_comm-release-release-indigo-rosconsole-1.11.16-0
- tar:
    local-name: ros_comm/roscpp
    uri: https://github.com/ros-gbp/ros_comm-release/archive/release/indigo/roscpp/1.11.16-0.tar.gz
    version: ros_comm-release-release-indigo-roscpp-1.11.16-0
- tar:
    local-name: ros_comm/xmlrpcpp
    uri: https://github.com/ros-gbp/ros_comm-release/archive/release/indigo/xmlrpcpp/1.11.16-0.tar.gz
    version: ros_comm-release-release-indigo-xmlrpcpp-1.11.16-0
- tar:
    local-name: ros_comm_msgs/rosgraph_msgs
    uri: https://github.com/ros-gbp/ros_comm_msgs-release/archive/release/indigo/rosgraph_msgs/1.11.1-0.tar.gz
    version: ros_comm_msgs-release-release-indigo-rosgraph_msgs-1.11.1-0
- tar:
    local-name: roscpp_core/cpp_common
    uri: https://github.com/ros-gbp/roscpp_core-release/archive/release/indigo/cpp_common/0.5.6-0.tar.gz
    version: roscpp_core-release-release-indigo-cpp_common-0.5.6-0
- tar:
    local-name: roscpp_core/roscpp_serialization
    uri: https://github.com/ros-gbp/roscpp_core-release/archive/release/indigo/roscpp_serialization/0.5.6-0.tar.gz
    version: roscpp_core-release-release-indigo-roscpp_serialization-0.5.6-0
- tar:
    local-name: roscpp_core/roscpp_traits
    uri: https://github.com/ros-gbp/roscpp_core-release/archive/release/indigo/roscpp_traits/0.5.6-0.tar.gz
    version: roscpp_core-release-release-indigo-roscpp_traits-0.5.6-0
- tar:
    local-name: roscpp_core/rostime
    uri: https://github.com/ros-gbp/roscpp_core-release/archive/release/indigo/rostime/0.5.6-0.tar.gz
    version: roscpp_core-release-release-indigo-rostime-0.5.6-0
- tar:
    local-name: rospack
    uri: https://github.com/ros-gbp/rospack-release/archive/release/indigo/rospack/2.2.5-0.tar.gz
    version: rospack-release-release-indigo-rospack-2.2.5-0
- tar:
    local-name: std_msgs
    uri: https://github.com/ros-gbp/std_msgs-release/archive/release/indigo/std_msgs/0.5.9-1.tar.gz
    version: std_msgs-release-release-indigo-std_msgs-0.5.9-1
</details>
comment
created
gavanderhoorn
username_0
Thanks @username_1.

re: did not take long: that is good to hear and was one of the reasons I wanted to improve the script. Notice how it skips generating the cache because it already exists.

The output also looks like I would have expected it.
comment
created
gavanderhoorn
username_0
O and one thing: if/when you have generated a few `.rosinstall` files -- and the accompanying `rosdistro` caches -- could you open a PR against [username_0/rgtm_rosdistro_caches](https://github.com/username_0/rgtm_rosdistro_caches) contributing the new caches.

That way things will go faster for everyone.
comment
created
git-afsantos
username_1
Not sure if I should open a new issue, of if it is intended behaviour (for now), and I just missed it.

When the generator needs to build a new cache, output redirection should be avoided - or else, you do not see the prompts for github credentials, which require you to hit Enter.

So the best workflow for now is to run once without redirection, to build the cache if needed, and then run again, redirecting to the intended file.
comment
created
gavanderhoorn
username_0
As also mentioned in https://github.com/robust-rosin/robust/issues/56#issuecomment-438438255, redirection should not be affecting the git pw prompts, as they should be appearing on `stderr`, which is not redirected with a regular `>`.

I'll see if I can figure out what is going on.
comment
created
gavanderhoorn
username_0
@username_1: I've added c7f97015. That appeared to be missing the redirect to `stderr`. A test on my machine with:

```
rosinstall_generator_tm.sh "2014-05-11T00:00:00+00:00" hydro > blaa.txt
```

shows all the output as I expected, including the pw and user prompts.
comment
created
git-afsantos
username_1
This works for me too. :+1:
comment
created
ChrisTimperley
username_2
I'm experiencing some unexpected behaviour with this version of the time machine:

```
building rosinstall file for file: mavros/101c09b/101c09b.bug
executing command: rosinstall_generator_tm.sh 2013-06-30 groovy mavros --deps --tar
time machine failed for bug [mavros/101c09b/101c09b.bug]:
Requested timepoint: '2013-06-30' (1372564800)
Switching to pre-REP-141 infrastructure ..
Resetting local rosdistro clone ..
Previous HEAD position was d0158a278... Merge pull request #1148 from 130s/bloom-patch-4
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
Determined rosdistro commit: d0158a27 (authored: 1372611034)
Reverting to ros/rosdistro@d0158a27
Cache already exists for (distro; stamp) tuple, skipping generation
Updating local rosdistro index.yaml to use cache from the past ..
Invoking: rosinstall_generator --rosdistro=groovy mavros --deps --tar
Using ROS_DISTRO: groovy
The following not released packages/stacks will be ignored: mavros
No packages/stacks left after ignoring not released
```
comment
created
gavanderhoorn
username_0
This is documented [here](https://github.com/rosin-project/rosinstall_generator_time_machine#no-packagesstacks-left-after-ignoring-not-released), but it basically means that `mavros` was not released in `groovy` at the time you requested.

Note btw: the code expects a ISO 8601 timestamp, which I don't believe `2013-06-30` is. Is this the json/Python parsing of the yaml again?

`date` and `git` can be flexible when parsing date-times, but just to avoid possible issues I would *really* recommend to use iso8601 stamps. They are much less ambiguous: `2016-06-30` does not include any info on timezones, which will make datetime parsing dependent on local time+date configuration.
comment
created
ChrisTimperley
username_2
comment
created
gavanderhoorn
username_0
Are you not using the `get_issue_creation_date.py` script? I would not use the datetimes in the `.bug` files. They are too imprecise. Bugs can be very dependent on the state of the pkgs (and thus on the state of `rosdistro`). Afaik, @wasowski retrieved most of those stamps by looking at the date that github renders on mouse-over.
comment
created
gavanderhoorn
username_0
O it could of course be that something is not working correctly. That's why it is a `WIP`.

If you'd manually checkout ros/rosdistro@d0158a27 you could see whether it's the correct state and whether `mavros` is listed in the relevant yaml files.
comment
created
gavanderhoorn
username_0
Just did this (through Github, just click the link) and I cannot find `mavros` in https://github.com/ros/rosdistro/blob/d0158a278ddee40b3408ec4111bdc777cf62803f/groovy/release.yaml.
comment
created
gavanderhoorn
username_0
I think I'm about ready to merge this PR.

Migrating everything to be included in the Docker image is something I'll do in a separate PR.

Things work sufficiently well for me right now with this new implementation.

@username_2, @username_1: would you have any comments?
comment
created
git-afsantos
username_1
:+1: I have not run into any issues since the one concerning output redirection.
comment
created
gavanderhoorn
username_0
Thanks for the "review" @username_1 :)
comment