Repository: ansible/ansible

Issue:
error building RPM re manpage 3 content
The "make rpm" command is failing because of errors on file
ansible / packaging / rpm /ansible.spec 
To make it working I had to comment lines 45 and 60.



Discussion:
Already fixed by #1262.

BTW, "pip install argparse" is doable on older platforms and will work around this problem w/o the pull request.

We're still going to fix it though, as I understand not everyone knows about that and distros are probably not going to package argparse.

Please let me know when the fix will be released because the current version is still broken.

Nothing is broken, you are just missing a build dependency and the pull
request has not been tested yet.

easy_install pip
pip install argparse

Simple!

On Wed, Oct 10, 2012 at 10:46 AM, Nicola Asuni notifications@github.comwrote:

> Please let me know when the fix will be released because the current
> version is still broken.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/ansible/ansible/issues/1277#issuecomment-9305591.

3 days ago the make rpm was working and now is not working anymore. I checked the ansible.spec and I commenting out two lines related to documentation dirs, the system is working again. Are we talking about the same problem?

I have installed the missing packages but I am getting the same error:
- cp -v 'docs/man/man3/_.3' /home/makerpm/ansible/rpm-build/BUILDROOT/ansible-0.8-0.git201210101234.el6.x86_64 //usr/share/man/man3/
  cp: cannot stat `docs/man/man3/_.3': No such file or directory
  error: Bad exit status from /var/tmp/rpm-tmp.nKYcv9 (%install)
  
  RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.nKYcv9 (%install)
  make: **\* [rpm] Error 1

This because the ansible.spec is not yet fixed.

NOTE: this bug is not related to #1262.

Dag is talking about something else, which confused me also.

Please paste the actual error message you are seeing from the latest git.

Nevermind, saw your last post, thanks

I think I have some unpushed changes on my computer -- will investigate later.  This is a result of merging the repos.

Look at the patch! The fix is in the patch linked to that PR. The fix is **both** a change to the Makefile to include the docs-target in the sdist-target as well as a fix to the _hacking/module_formatter.py_ script so the man3 pages are in fact created and part of the tarball that is used by the SPEC file.

Unless of course you do not want the man3 pages in the SPEC file, but what's the purpose of having these man-pages if they are not being part of the package ?

I do want them in.

Merged stuff in, but currently getting this:

Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/mdehaan/ansible/rpm-build/BUILDROOT/ansible-0.8-0.git201210110009.el6.x86_64
error: Installed (but unpackaged) file(s) found:
   /home/mdehaan/ansible/library/apt
   /home/mdehaan/ansible/library/apt_repository
   /home/mdehaan/ansible/library/assemble
   /home/mdehaan/ansible/library/async_status
   /home/mdehaan/ansible/library/async_wrapper
   /home/mdehaan/ansible/library/authorized_key
   /home/mdehaan/ansible/library/command

Any ideas @dagwieers or @tbielawa ?

Thanks!!!

@nicolafubra @mpdehaan I'm checking this out now. But first I must run to the store for beer.

Unfortunately in my initial attempts I could not reproduce this bug. Build result here: http://koji.fedoraproject.org/koji/taskinfo?taskID=4580033

Built from 6d2f003cebf77df3cdff19a70c2fefd7bd890abf using `make srpm`. Sent srpm to fedorahosted EL6 koji builders (which run mock) and received an RPM. http://kojipkgs.fedoraproject.org//work/tasks/33/4580033/ansible-0.8-0.git201210110125.el6.noarch.rpm

@mpdehaan @nicolafubra @dagwieers Can one or more of you (who can reproduce) please provide:
- `git log -1` to verify we're testing the same commit
- `cat /etc/redhat-release` (or something comparable) to verify we're testing the same build target
- The **complete** log from `make srpm` and `make rpm`

```
<tbielawa>@(deepfryer)[~/Projects/ansible] 21:30:29  (devel) 
$ make srpm
# whole bunch of stuff
hard linking packaging/rpm/ansible.spec -> ansible-0.8/packaging/rpm
creating dist
Creating tar archive
removing 'ansible-0.8' (and everything under it)
Wrote: /home/tbielawa/Projects/ansible/rpm-build/ansible-0.8-0.git201210110125.fc17.src.rpm
#############################################
Ansible SRPM is built:
    rpm-build/ansible-0.8-0.git201210110125.fc17.src.rpm
#############################################

<tbielawa>@(deepfryer)[~/Projects/ansible] 21:30:49  (devel) 
$ koji build --scratch dist-6E-epel ./rpm-build/ansible-0.8-0.git201210110125.fc17.src.rpm   
Uploading srpm: ./rpm-build/ansible-0.8-0.git201210110125.fc17.src.rpm
[====================================] 100% 00:00:01 174.70 KiB 102.84 KiB/sec
Created task: 4580032
Task info: http://koji.fedoraproject.org/koji/taskinfo?taskID=4580032
Watching tasks (this may be safely interrupted)...
4580032 build (dist-6E-epel, ansible-0.8-0.git201210110125.fc17.src.rpm): free
4580032 build (dist-6E-epel, ansible-0.8-0.git201210110125.fc17.src.rpm): free -> open (buildvm-29.phx2.fedoraproject.org)
  4580033 buildArch (ansible-0.8-0.git201210110125.fc17.src.rpm, noarch): free
  4580033 buildArch (ansible-0.8-0.git201210110125.fc17.src.rpm, noarch): free -> open (buildvm-20.phx2.fedoraproject.org)
  4580033 buildArch (ansible-0.8-0.git201210110125.fc17.src.rpm, noarch): open (buildvm-20.phx2.fedoraproject.org) -> closed
  0 free  1 open  1 done  0 failed
4580032 build (dist-6E-epel, ansible-0.8-0.git201210110125.fc17.src.rpm): open (buildvm-29.phx2.fedoraproject.org) -> closed
  0 free  0 open  2 done  0 failed

4580032 build (dist-6E-epel, ansible-0.8-0.git201210110125.fc17.src.rpm) completed successfully


<tbielawa>@(deepfryer)[~/Projects/ansible] 21:37:06  (devel) 
$ git log -1
commit 6d2f003cebf77df3cdff19a70c2fefd7bd890abf
Author: Michael DeHaan <michael.dehaan@gmail.com>
Date:   Wed Oct 10 21:25:19 2012 -0400

    Update formatter to not include orange blocks all over docsite page
```

I have also tested using mock to build the SRPM and then the subsequent RPM:

```
<tbielawa>@(deepfryer)[~/Projects/ansible] 22:02:35  (devel) 
$ ~/bin/ansible-mockbuild-SRPM-test.sh > ~/ansible-SRPM-epel-6-x86_64-mockbuild.log 2>&1
...
<tbielawa>@(deepfryer)[~/Projects/ansible] 22:08:45  (devel)
$ ~/bin/ansible-mockbuild-RPM-test.sh > ~/ansible-RPM-epel-6-x86_64-mockbuild.log 2>&1
...
<tbielawa>@(deepfryer)[~/Projects/ansible] 22:10:33  (devel)
$ rpmfile /var/lib/mock/epel-6-x86_64/result/ansible-0.8-1.el6.noarch.rpm > ~/ansible-RPM-epel-6-x86_64-rpmfile-contents.log
```

The results are pretty long, so here they are linked offsite:

http://lnx.cx/~tbielawa/ansible-SRPM-epel-6-x86_64-mockbuild.log

http://lnx.cx/~tbielawa/ansible-RPM-epel-6-x86_64-mockbuild.log

http://lnx.cx/~tbielawa/ansible-RPM-epel-6-x86_64-rpmfile-contents.log

Tip also and CentOS 6.2 for me.  No user rpm macro config or anything.  Will get logs.

Sent from my iPad

On Oct 10, 2012, at 10:12 PM, tbielawa notifications@github.com wrote:

> Unfortunately in my initial attempts I could not reproduce this bug. Build result here: http://koji.fedoraproject.org/koji/taskinfo?taskID=4580033
> 
> Built from 6d2f003 using make srpm. Sent srpm to fedorahosted EL6 koji builders (which run mock) and received an RPM. http://kojipkgs.fedoraproject.org//work/tasks/33/4580033/ansible-0.8-0.git201210110125.el6.noarch.rpm
> 
> @mpdehaan @nicolafubra @dagwieers Can one or more of you (who can reproduce) please provide:
> 
> git log -1 to verify we're testing the same commit
> cat /etc/redhat-release (or something comparable) to verify we're testing the same build target
> The complete log from make srpm and make rpm
> <tbielawa>@(deepfryer)[~/Projects/ansible] 21:30:29  (devel) 
> $ make srpm
> 
> # whole bunch of stuff
> 
> hard linking packaging/rpm/ansible.spec -> ansible-0.8/packaging/rpm
> creating dist
> Creating tar archive
> removing 'ansible-0.8' (and everything under it)
> Wrote: /home/tbielawa/Projects/ansible/rpm-build/ansible-0.8-0.git201210110125.fc17.src.rpm
> #############################################
> Ansible SRPM is built:
>     rpm-build/ansible-0.8-0.git201210110125.fc17.src.rpm
> #############################################
> 
> <tbielawa>@(deepfryer)[~/Projects/ansible] 21:30:49  (devel) 
> $ koji build --scratch dist-6E-epel ./rpm-build/ansible-0.8-0.git201210110125.fc17.src.rpm  
> Uploading srpm: ./rpm-build/ansible-0.8-0.git201210110125.fc17.src.rpm
> [====================================] 100% 00:00:01 174.70 KiB 102.84 KiB/sec
> Created task: 4580032
> Task info: http://koji.fedoraproject.org/koji/taskinfo?taskID=4580032
> Watching tasks (this may be safely interrupted)...
> 4580032 build (dist-6E-epel, ansible-0.8-0.git201210110125.fc17.src.rpm): free
> 4580032 build (dist-6E-epel, ansible-0.8-0.git201210110125.fc17.src.rpm): free -> open (buildvm-29.phx2.fedoraproject.org)
>   4580033 buildArch (ansible-0.8-0.git201210110125.fc17.src.rpm, noarch): free
>   4580033 buildArch (ansible-0.8-0.git201210110125.fc17.src.rpm, noarch): free -> open (buildvm-20.phx2.fedoraproject.org)
>   4580033 buildArch (ansible-0.8-0.git201210110125.fc17.src.rpm, noarch): open (buildvm-20.phx2.fedoraproject.org) -> closed
>   0 free  1 open  1 done  0 failed
> 4580032 build (dist-6E-epel, ansible-0.8-0.git201210110125.fc17.src.rpm): open (buildvm-29.phx2.fedoraproject.org) -> closed
>   0 free  0 open  2 done  0 failed
> 
> 4580032 build (dist-6E-epel, ansible-0.8-0.git201210110125.fc17.src.rpm) completed successfully
> 
> <tbielawa>@(deepfryer)[~/Projects/ansible] 21:37:06  (devel) 
> $ git log -1
> commit 6d2f003cebf77df3cdff19a70c2fefd7bd890abf
> Author: Michael DeHaan michael.dehaan@gmail.com
> Date:   Wed Oct 10 21:25:19 2012 -0400
> 
> ```
> Update formatter to not include orange blocks all over docsite page
> ```
> 
> I have also tested using mock to build the SRPM and then the subsequent RPM:
> 
> <tbielawa>@(deepfryer)[~/Projects/ansible] 22:02:35  (devel) 
> $ ~/bin/ansible-mockbuild-SRPM-test.sh > ~/ansible-SRPM-epel-6-x86_64-mockbuild.log 2>&1
> ...
> <tbielawa>@(deepfryer)[~/Projects/ansible] 22:08:45  (devel)
> $ ~/bin/ansible-mockbuild-RPM-test.sh > ~/ansible-RPM-epel-6-x86_64-mockbuild.log 2>&1
> ...
> <tbielawa>@(deepfryer)[~/Projects/ansible] 22:10:33  (devel)
> $ rpmfile /var/lib/mock/epel-6-x86_64/result/ansible-0.8-1.el6.noarch.rpm > ~/ansible-RPM-epel-6-x86_64-rpmfile-contents.log
> The results are pretty long, so here they are linked offsite:
> 
> http://lnx.cx/~tbielawa/ansible-SRPM-epel-6-x86_64-mockbuild.log
> 
> http://lnx.cx/~tbielawa/ansible-RPM-epel-6-x86_64-mockbuild.log
> 
> http://lnx.cx/~tbielawa/ansible-RPM-epel-6-x86_64-rpmfile-contents.log
> 
> —
> Reply to this email directly or view it on GitHub.

Works fine for me on RHEL6.3:

```
#############################################
Ansible RPM is built:
    rpm-build/ansible-0.8-0.git201210110125.el6.noarch.rpm
#############################################
```

Now the situation seems worse than before:

[root@rpmbuild makerpm]# su -c 'cd /home/makerpm && git clone git://github.com/ansible/ansible.git' makerpm
Initialized empty Git repository in /home/makerpm/ansible/.git/
remote: Counting objects: 15960, done.
remote: Compressing objects: 100% (4325/4325), done.
remote: Total 15960 (delta 10593), reused 15721 (delta 10506)
Receiving objects: 100% (15960/15960), 3.83 MiB | 1.93 MiB/s, done.
Resolving deltas: 100% (10593/10593), done.

[root@rpmbuild ansible]# git log -1
commit 6d2f003cebf77df3cdff19a70c2fefd7bd890abf
Author: Michael DeHaan michael.dehaan@gmail.com
Date:   Wed Oct 10 21:25:19 2012 -0400

[root@rpmbuild ansible]# cat /etc/redhat-release
CentOS release 6.3 (Final)

[root@rpmbuild ansible]# uname -a
Linux rpmbuild.localdomain 2.6.32-279.9.1.el6.x86_64 #1 SMP Tue Sep 25 21:43:11 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

[root@rpmbuild makerpm]# su -c 'cd /home/makerpm/ansible && make rpm' makerpm
Cleaning up distutils stuff
rm -rf build
rm -rf dist
Cleaning up byte compiled python stuff
find . -type f -regex "._.py[co]$" -delete
Cleaning up editor backup files
find . -type f ( -name "_~" -or -name "#_" ) -delete
find . -type f ( -name "_.swp" ) -delete
Cleaning up manpage stuff
find ./docs/man -type f -name "_.xml" -delete
find ./docs/man -type f -name "_.asciidoc" -delete
find ./docs/man/man3 -type f -name "_.3" -delete
Cleaning up output from test runs
rm -rf test/test_data
Cleaning up RPM building stuff
rm -rf MANIFEST rpm-build
Cleaning up Debian building stuff
rm -rf debian
rm -rf deb-build
rm -rf docs/json
rm -rf docs/js
sed "s/%VERSION%/0.8/" docs/man/man1/ansible-playbook.1.asciidoc.in > docs/man/man1/ansible-playbook.1.asciidoc
a2x -D docs/man/man1/ -d manpage -f manpage docs/man/man1/ansible-playbook.1.asciidoc
make: execvp: a2x: Permission denied
make: *_\* [docs/man/man1/ansible-playbook.1] Error 127
rm docs/man/man1/ansible-playbook.1.asciidoc

This bug seems now fixed:
#############################################
Ansible RPM is built:
    rpm-build/ansible-0.8-0.git201210110125.el6.noarch.rpm
#############################################

> (12:47:28 PM) nckans: the new version of Ansible requires asciidoc? if yes it should be added on dependencies and work anyway
> (12:48:44 PM) nckans: If particular packages are required to build Ansible, then this must be listed somewhere or better added on spec file

Ansible uses asciidoc to generate a working tarball with documentation included, because of this it should **not** be/become a _BuilldRequires_ to the RPM package.

Correct, asciidoc generated things are checked in to source control.

I still see "make rpm" on my environment complaining about modules that are not installed, however, but this appears to be only happening to me and not in Fedora, so... perhaps something I just need to figure out :)

RPM build errors:
    Installed (but unpackaged) file(s) found:
   /home/mdehaan/ansible/library/apt
   /home/mdehaan/ansible/library/apt_repository
   /home/mdehaan/ansible/library/assemble
   /home/mdehaan/ansible/library/async_status
   /home/mdehaan/ansible/library/async_wrapper
   /home/mdehaan/ansible/library/authorized_key
   /home/mdehaan/ansible/library/command
   /home/mdehaan/ansible/library/copy

etc

Reproduced with @dhozac's insight on IRC today. Setting `library` in the `ansible.cfg` file makes the build fail.

Well there's your problem:

```
# setup.py
...
eee2d1af (Michael DeHaan         2012-08-16 21:34:55 -0400 11) # find library modules
eee2d1af (Michael DeHaan         2012-08-16 21:34:55 -0400 12) from ansible.constants import DEFAULT_MODULE_PATH
eee2d1af (Michael DeHaan         2012-08-16 21:34:55 -0400 13) data_files = [ (DEFAULT_MODULE_PATH, glob('./library/*')) ]                                     
```

This is saying to install the contents of ./library into `DEFAULT_MODULE_PATH`. This is effectively how we support users having a directory of custom modules which is augmented with the upstream Ansible module library during install time.

What we're trying to resolve now is finding a method that (roughly) supports:
1. Users being able to set where the upstream ansible modules install to
2. The RPM  build not exploding when `DEFAULT_MODULE_PATH` is redefined and `%{_datadir}/ansible/` no longer represents `DEFAULT_MODULE_PATH`

Here's a diff that would fix things to just installing into /usr/share/ansible/:

```
diff --git a/lib/ansible/constants.py b/lib/ansible/constants.py
index dffcc24..c68a19f 100644
--- a/lib/ansible/constants.py
+++ b/lib/ansible/constants.py
@@ -60,12 +60,16 @@ p = load_config_file()

 active_user   = pwd.getpwuid(os.geteuid())[0]

+# Needed so the RPM can call setup.py and have modules land in the
+# correct location.
+DIST_MODULE_PATH = '/usr/share/ansible/'
+
 # sections in config file
 DEFAULTS='defaults'

 # configurable things
 DEFAULT_HOST_LIST         = shell_expand_path(get_config(p, DEFAULTS, 'hostfile',         'ANSIBLE_HOSTS',            '/etc/ansible/hosts'))
-DEFAULT_MODULE_PATH       = shell_expand_path(get_config(p, DEFAULTS, 'library',          'ANSIBLE_LIBRARY',          '/usr/share/ansible'))
+DEFAULT_MODULE_PATH       = shell_expand_path(get_config(p, DEFAULTS, 'library',          'ANSIBLE_LIBRARY',          DIST_MODULE_PATH))
 DEFAULT_REMOTE_TMP        = shell_expand_path(get_config(p, DEFAULTS, 'remote_tmp',       'ANSIBLE_REMOTE_TEMP',      '$HOME/.ansible/tmp'))
 DEFAULT_MODULE_NAME       = get_config(p, DEFAULTS, 'module_name',      None,                       'command')
 DEFAULT_PATTERN           = get_config(p, DEFAULTS, 'pattern',          None,                       '*')
diff --git a/setup.py b/setup.py
index 8d76891..06f357b 100644
--- a/setup.py
+++ b/setup.py
@@ -9,8 +9,8 @@ from ansible import __version__, __author__
 from distutils.core import setup

 # find library modules
-from ansible.constants import DEFAULT_MODULE_PATH
-data_files = [ (DEFAULT_MODULE_PATH, glob('./library/*')) ]
+from ansible.constants import DIST_MODULE_PATH
+data_files = [ (DIST_MODULE_PATH, glob('./library/*')) ]

 print "DATA FILES=%s" % data_files
```

But in a way it's pointless, because ansible modules will go into _location1_ while when ansible runs it will look in _location2_ (set via `~/.ansible.cfg`)

yep, I think we can safely replace this with os.getcwd() versus the
constants stuff.

On Thu, Oct 11, 2012 at 12:41 PM, tbielawa notifications@github.com wrote:

> Reproduced with @dhozac https://github.com/dhozac's insight on IRC
> today. Setting library in the ansible.cfg file makes the build fail.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/ansible/ansible/issues/1277#issuecomment-9348110.

It's crazy simpler than that patch, actually.

Make setup.py just look at os.getcwd() and not constants.py

An application should never run live code from the application to build
itself.

hmm, nm, I think you are correct, it's just about the constant, not
the other.  I'm confused.

yes.


Resolving patch:
diff --git a/lib/ansible/constants.py b/lib/ansible/constants.py
--- a/lib/ansible/constants.py
+++ b/lib/ansible/constants.py
@@ -60,12 +60,16 @@ def shell_expand_path(path):
 
 active_user   = pwd.getpwuid(os.geteuid())[0]
 
+# Needed so the RPM can call setup.py and have modules land in the
+# correct location. See #1277 for discussion
+DIST_MODULE_PATH = '/usr/share/ansible/'
+
 # sections in config file
 DEFAULTS='defaults'
 
 # configurable things
 DEFAULT_HOST_LIST         = shell_expand_path(get_config(p, DEFAULTS, 'hostfile',         'ANSIBLE_HOSTS',            '/etc/ansible/hosts'))
-DEFAULT_MODULE_PATH       = shell_expand_path(get_config(p, DEFAULTS, 'library',          'ANSIBLE_LIBRARY',          '/usr/share/ansible'))
+DEFAULT_MODULE_PATH       = shell_expand_path(get_config(p, DEFAULTS, 'library',          'ANSIBLE_LIBRARY',          DIST_MODULE_PATH))
 DEFAULT_REMOTE_TMP        = shell_expand_path(get_config(p, DEFAULTS, 'remote_tmp',       'ANSIBLE_REMOTE_TEMP',      '$HOME/.ansible/tmp'))
 DEFAULT_MODULE_NAME       = get_config(p, DEFAULTS, 'module_name',      None,                       'command')
 DEFAULT_PATTERN           = get_config(p, DEFAULTS, 'pattern',          None,                       '*')
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -9,8 +9,8 @@
 from distutils.core import setup
 
 # find library modules
-from ansible.constants import DEFAULT_MODULE_PATH
-data_files = [ (DEFAULT_MODULE_PATH, glob('./library/*')) ]
+from ansible.constants import DIST_MODULE_PATH
+data_files = [ (DIST_MODULE_PATH, glob('./library/*')) ]
 
 print "DATA FILES=%s" % data_files
 


Test patch: