mozy/ruby-protocol-buffers — Issue #16: File creation of rprotoc doesn't match behavior of protoc
Author: sahuagin
Created: 2011-12-28 15:48:02+00:00
State: open

Related to issues #9, #10.

This relates to proto files that imports other proto files. My specific use case involves having a directory hierarchy that matches the namespace hierarchy. A short solution would be for rprotoc to only output the proto file it's being asked to compile and not also generate all imports at the same time.

Initially I ran rprotoc and kept all of the files in a top level directory. Then imports fail because they are structured as 'namespace/File.pb.rb'. Copying the output into the same directory structure would give me runtime errors since both versions of the pb.rb file were being loaded.

I've found a few work arounds. Currently I'm creating the directory hierarchy and running multiple compiles, one for each file. Side effect is redundant files which I will then go and delete.

I'm currently working on Mac OS X with ruby 1.8.7, FreeBSD 8.2 with ruby 1.9 and 1.8.7. ruby_protobuf gem is 0.4.11.

Included below is the commands and output from rprotoc and protoc. Output is from FreeBSD 8.2, ruby 1.8.7, ruby_protobuf 0.4.11, protoc version 2.4.1

```shell

Related to issues #9, #10.

Directory structure.
code/gbe/pnl

$ ls -alR gbe
total 30
drwxr-xr-x 3 fuzzle fizzle 5 Dec 28 10:58 .
drwxr-xr-x 5 fuzzle fizzle 16 Dec 28 10:53 ..
-rw-r--r-- 1 fuzzle fizzle 3913 Dec 28 10:50 Portfolio.proto
drwxr-xr-x 2 fuzzle fizzle 3 Dec 28 10:57 pnl

gbe/pnl:
total 6
drwxr-xr-x 2 fuzzle fizzle 3 Dec 28 10:57 .
drwxr-xr-x 3 fuzzle fizzle 5 Dec 28 10:58 ..
-rw-r--r-- 1 fuzzle fizzle 2149 Dec 28 10:48 Statistics.proto

# run the compiler on the top level proto file
$ rprotoc -o gbe -p ".;gbe" gbe/Portfolio.proto
gbe/Statistics.pb.rb writing...
Warning, couldn't test load proto file because of imports
gbe/Portfolio.pb.rb writing...
$ ls -alR gbe
total 31
drwxr-xr-x 3 fuzzle fizzle 7 Dec 28 10:59 .
drwxr-xr-x 5 fuzzle fizzle 16 Dec 28 10:53 ..
-rw-r--r-- 1 fuzzle fizzle 6505 Dec 28 10:59 Portfolio.pb.rb
-rw-r--r-- 1 fuzzle fizzle 3913 Dec 28 10:50 Portfolio.proto
-rw-r--r-- 1 fuzzle fizzle 3864 Dec 28 10:59 Statistics.pb.rb
drwxr-xr-x 2 fuzzle fizzle 3 Dec 28 10:57 pnl

gbe/pnl:
total 6
drwxr-xr-x 2 fuzzle fizzle 3 Dec 28 10:57 .
drwxr-xr-x 3 fuzzle fizzle 7 Dec 28 10:59 ..
-rw-r--r-- 1 fuzzle fizzle 2149 Dec 28 10:48 Statistics.proto
$

# but, now loads won't work because the import statement is:
import 'gbe/pnl/Statistics.proto';

$ ruby18 ./Portfolio.rb | less
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require': no such file to load -- gbe/pnl/Statistics.pb (LoadError)
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
from ./gbe/Portfolio.pb.rb:155
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require'
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
from ./Portfolio.rb:2

# so, now go run the compiler on the sub-directory

$ rprotoc -o gbe/pnl -p ".;gbe" gbe/pnl/Statistics.proto
gbe/pnl/Statistics.pb.rb writing...
$ ls -alR gbe
total 41
drwxr-xr-x 3 fuzzle fizzle 7 Dec 28 10:59 .
drwxr-xr-x 5 fuzzle fizzle 16 Dec 28 10:53 ..
-rw-r--r-- 1 fuzzle fizzle 6505 Dec 28 10:59 Portfolio.pb.rb
-rw-r--r-- 1 fuzzle fizzle 3913 Dec 28 10:50 Portfolio.proto
-rw-r--r-- 1 fuzzle fizzle 3864 Dec 28 10:59 Statistics.pb.rb
drwxr-xr-x 2 fuzzle fizzle 4 Dec 28 11:02 pnl

gbe/pnl:
total 7
drwxr-xr-x 2 fuzzle fizzle 4 Dec 28 11:02 .
drwxr-xr-x 3 fuzzle fizzle 7 Dec 28 10:59 ..
-rw-r--r-- 1 fuzzle fizzle 3864 Dec 28 11:02 Statistics.pb.rb
-rw-r--r-- 1 fuzzle fizzle 2149 Dec 28 10:48 Statistics.proto

# now the program will run, if done as this example
$ ruby18 ./Portfolio.rb
...correct output...
$

# but, now there are multiple Statistics outputs located in the build system. Some imports will fail because they find both Statistics.pb.rb files
# and the numbers collide.

##### Using the protoc compiler ######

# the initial directory layout
$ ls -alR gbe
total 30
drwxr-xr-x 3 fuzzle fizzle 5 Dec 28 11:05 .
drwxr-xr-x 5 fuzzle fizzle 16 Dec 28 10:53 ..
-rw-r--r-- 1 fuzzle fizzle 20480 Dec 28 10:51 .Portfolio.proto.swp
-rw-r--r-- 1 fuzzle fizzle 3913 Dec 28 10:50 Portfolio.proto
drwxr-xr-x 2 fuzzle fizzle 3 Dec 28 11:05 pnl

gbe/pnl:
total 6
drwxr-xr-x 2 fuzzle fizzle 3 Dec 28 11:05 .
drwxr-xr-x 3 fuzzle fizzle 5 Dec 28 11:05 ..
-rw-r--r-- 1 fuzzle fizzle 2149 Dec 28 10:48 Statistics.proto
[fuzzle@gbe-jamdev-02 ~/work/pjm/sample_code/ruby_protobuf]$

# running protoc
$ protoc --cpp_out=. --proto_path . --proto_path gbe gbe/Portfolio.proto
$ ls -alR gbe
total 31
drwxr-xr-x 3 fuzzle fizzle 7 Dec 28 11:11 .
drwxr-xr-x 5 fuzzle fizzle 16 Dec 28 10:53 ..
-rw-r--r-- 1 fuzzle fizzle 116777 Dec 28 11:11 Portfolio.pb.cc
-rw-r--r-- 1 fuzzle fizzle 76031 Dec 28 11:11 Portfolio.pb.h
-rw-r--r-- 1 fuzzle fizzle 3913 Dec 28 10:50 Portfolio.proto
drwxr-xr-x 2 fuzzle fizzle 3 Dec 28 11:05 pnl

gbe/pnl:
total 6
drwxr-xr-x 2 fuzzle fizzle 3 Dec 28 11:05 .
drwxr-xr-x 3 fuzzle fizzle 7 Dec 28 11:11 ..
-rw-r--r-- 1 fuzzle fizzle 2149 Dec 28 10:48 Statistics.proto

# NOTE: Above in the directory structure we were putting the output files into . and they ended up in gbe. The protoc
# compiler implicitly assumes that you'll have a directory structure similar to the namespace.
# Also, there is only the output for the file that we asked it to compile, not the file + imports.

# running the command on the statistics file.
$ protoc --cpp_out=. --proto_path . --proto_path gbe gbe/pnl/Statistics.proto
$ ls -alR gbe
total 220
drwxr-xr-x 3 fuzzle fizzle 7 Dec 28 11:11 .
drwxr-xr-x 5 fuzzle fizzle 16 Dec 28 10:53 ..
-rw-r--r-- 1 fuzzle fizzle 116777 Dec 28 11:11 Portfolio.pb.cc
-rw-r--r-- 1 fuzzle fizzle 76031 Dec 28 11:11 Portfolio.pb.h
-rw-r--r-- 1 fuzzle fizzle 3913 Dec 28 10:50 Portfolio.proto
drwxr-xr-x 2 fuzzle fizzle 5 Dec 28 11:13 pnl

gbe/pnl:
total 7
drwxr-xr-x 2 fuzzle fizzle 5 Dec 28 11:13 .
drwxr-xr-x 3 fuzzle fizzle 7 Dec 28 11:11 ..
-rw-r--r-- 1 fuzzle fizzle 55621 Dec 28 11:13 Statistics.pb.cc
-rw-r--r-- 1 fuzzle fizzle 37536 Dec 28 11:13 Statistics.pb.h
-rw-r--r-- 1 fuzzle fizzle 2149 Dec 28 10:48 Statistics.proto

# now the files are created and can be imported.
# NOTE: we asked for the output to be in '.' and the Statistics files ended up in gbe/pnl which matches the namespace and our import statements.

# Similar output is obtained if you run using python_out.

```