Repository: ycm-core/ycmd

Issue:
Implement GoTo for Go
Backstory: After I had just fixed up the Sublime plugin to work with Rust, it turned out that I would be reading a lot of Go code for a while and I didn't actually had a chance to play with YCM for Rust much even though I usually program in it. The universe is cruel :unamused: 

I miss GoTo a lot in Go (doubly so than Rust since the docs are less usable for me), and while I tried fixing up gocode to have GoTo, there are some core issues with its design which make this hard.

Anyway, someone pointed me to https://github.com/rogpeppe/godef, which I'd like to integrate. At first glance it seems easy; but that's what they always say :laughing: 

I plan (at least for now) to do the work here; just opening this issue so that I can post notes/ask for help and in case you don't actually want this integration :smile: 

Tips welcome!



Discussion:
Thanks for expressing interest in working on ycmd! :)

Hm, having two engines (one for completion, the other for GoTo) for separate parts of a language's semantic support would be a first for us. 

@micbou @puremourning @vheon @ekfriis Thoughts?

Yeah, that's one issue I expected.

Note that godef is a simple binary, without a server. (Gocode is also a simple binary, but it internally does cool stuff with spinning up a server and using unix sockets or something to communicate). Nor does godef _need_ a server, since GoTo perf isn't that important as long as it's within a few hundred milliseconds (unlike autocomplete which should be buttery smooth).

IMO it should be fine to call a different standalone binary for goto.

Also, godef seems to mostly let the Go AST/parse libraries do the heavy lifting, though it contains a copy of them for some reason (backcompat probably)

The only issue is that our build system becomes more complicated; we bundle and build _everything_ for the user. This being Go, it shouldn't be much of an issue.

If godef is this simple, why not simply use it as a library from within gocode? I'm guessing upstream wouldn't be too happy about that idea, but might be worth asking. 

Forgot to mention: I'm personally OK with this idea. I'm not a _fan_ of it, but I can see how it would make users happy for not too much hackery. I would like to see how intrusive it would be code-wise though. (I don't expect this to be an issue.)

Other ycmd devs might have different insights.

The golang community follow the UNIX philosophy, so every functionality that you might want is actually a separate executable. For this reason I find this

> If godef is this simple, why not simply use it as a library from within gocode? I'm guessing upstream wouldn't be too happy about that idea, but might be worth asking.

to be unlikely accepted upstream. The downside that I see are:
- a "IDE like system" would reuse the AST for the code if it could.
- on our side we wouldn't spawn a new process every time we want a GoTo

On the other hand I don't see this as a drama as it would be for other languages:
- The Go language was designed to be easy to parse and to compile (removing also circular imports), so creating the AST for the source code is really fast.
- Being statically compiled we don't pay for the time of runtime linking which means that spawning a go tool is actually fast.

So I would say that it would be absolutely ok to integrate a different tool for different functionality.

P.S: I know that is not a good excuse but every other system supporting Go as a language would do exactly what we would do here.

Needs: https://github.com/rogpeppe/godef/pull/24


Resolving patch:
diff --git a/build.py b/build.py
--- a/build.py
+++ b/build.py
@@ -283,6 +283,8 @@ def BuildGoCode():
 
   os.chdir( p.join( DIR_OF_THIS_SCRIPT, 'third_party', 'gocode' ) )
   subprocess.check_call( [ 'go', 'build' ] )
+  os.chdir( p.join( DIR_OF_THIS_SCRIPT, 'third_party', 'godef' ) )
+  subprocess.check_call( [ 'go', 'build' ] )
 
 
 def BuildRacerd():
diff --git a/ycmd/completers/go/gocode_completer.py b/ycmd/completers/go/gocode_completer.py
--- a/ycmd/completers/go/gocode_completer.py
+++ b/ycmd/completers/go/gocode_completer.py
@@ -27,10 +27,10 @@
 from ycmd.completers.completer import Completer
 
 GO_FILETYPES = set( [ 'go' ] )
-BINARY_NOT_FOUND_MESSAGE = ( 'Gocode binary not found. Did you build it? ' +
+BINARY_NOT_FOUND_MESSAGE = ( '{0} binary not found. Did you build it? ' +
                              'You can do so by running ' +
                              '"./install.py --gocode-completer".' )
-COMPLETION_ERROR_MESSAGE = 'Gocode shell call failed.'
+SHELL_ERROR_MESSAGE = '{0} shell call failed.'
 PARSE_ERROR_MESSAGE = 'Gocode returned invalid JSON response.'
 NO_COMPLETIONS_MESSAGE = 'Gocode returned empty JSON response.'
 GOCODE_PANIC_MESSAGE = ( 'Gocode panicked trying to find completions, ' +
@@ -39,6 +39,10 @@
   os.path.abspath( os.path.dirname( __file__ ) ),
   '..', '..', '..', 'third_party', 'gocode',
   'gocode' + ( '.exe' if utils.OnWindows() else '' ) )
+PATH_TO_GODEF_BINARY = os.path.join(
+  os.path.abspath( os.path.dirname( __file__ ) ),
+  '..', '..', '..', 'third_party', 'godef',
+  'godef' + ( '.exe' if utils.OnWindows() else '' ) )
 
 _logger = logging.getLogger( __name__ )
 
@@ -48,13 +52,22 @@ class GoCodeCompleter( Completer ):
   def __init__( self, user_options ):
     super( GoCodeCompleter, self ).__init__( user_options )
     self._popener = utils.SafePopen # Overridden in test.
-    self._binary = self.FindGoCodeBinary( user_options )
+    self._binary_gocode = self.FindBinary( 'gocode', user_options )
+    self._binary_godef = self.FindBinary( 'godef', user_options )
+
+    if not self._binary_gocode:
+      _logger.error( BINARY_NOT_FOUND_MESSAGE.format( 'Gocode' ) )
+      raise RuntimeError( BINARY_NOT_FOUND_MESSAGE.format( 'Gocode' ) )
 
-    if not self._binary:
-      _logger.error( BINARY_NOT_FOUND_MESSAGE )
-      raise RuntimeError( BINARY_NOT_FOUND_MESSAGE )
+    _logger.info( 'Enabling go completion using %s binary',
+                  self._binary_gocode )
 
-    _logger.info( 'Enabling go completion using %s binary', self._binary )
+    if not self._binary_godef:
+      _logger.error( BINARY_NOT_FOUND_MESSAGE.format( 'Godef' ) )
+      raise RuntimeError( BINARY_NOT_FOUND_MESSAGE.format( 'Godef' ) )
+
+    _logger.info( 'Enabling go definitions using %s binary',
+                   self._binary_godef )
 
 
   def SupportedFiletypes( self ):
@@ -72,9 +85,10 @@ def ComputeCandidatesInner( self, request_data ):
     offset = _ComputeOffset( contents, request_data[ 'line_num' ],
                              request_data[ 'column_num' ] )
 
-    stdoutdata = self._ExecuteGoCodeBinary( '-f=json', 'autocomplete',
-                                            filename, str(offset),
-                                            contents = contents )
+    stdoutdata = self._ExecuteBinary( self._binary_gocode,
+                                      '-f=json', 'autocomplete',
+                                       filename, str(offset),
+                                       contents = contents )
 
     try:
       resultdata = json.loads( stdoutdata )
@@ -94,32 +108,42 @@ def ComputeCandidatesInner( self, request_data ):
   def GetSubcommandsMap( self ):
     return {
       'StartServer': ( lambda self, request_data, args: self._StartServer() ),
-      'StopServer': ( lambda self, request_data, args: self._StopServer() )
+      'StopServer': ( lambda self, request_data, args: self._StopServer() ),
+      'GoTo' : ( lambda self, request_data, args:
+                 self._GoToDefinition( request_data ) ),
+      'GoToDefinition' : ( lambda self, request_data, args:
+                           self._GoToDefinition( request_data ) ),
+      'GoToDeclaration' : ( lambda self, request_data, args:
+                           self._GoToDefinition( request_data ) ),
     }
 
 
-  def FindGoCodeBinary( self, user_options ):
-    """ Find the path to the gocode binary.
+  def FindBinary( self, binary, user_options ):
+    """ Find the path to the gocode/godef binary.
 
-    If 'gocode_binary_path' in the options is blank,
-    use the version installed with YCM, if it exists,
-    then the one on the path, if not.
+    If 'gocode_binary_path' or 'godef_binary_path'
+    in the options is blank, use the version installed
+    with YCM, if it exists, then the one on the path, if not.
 
-    If the 'gocode_binary_path' is specified, use it
-    as an absolute path.
+    If the 'gocode_binary_path' or 'godef_binary_path' is
+    specified, use it as an absolute path.
 
     If the resolved binary exists, return the path,
     otherwise return None. """
-    if user_options.get( 'gocode_binary_path' ):
+    if user_options.get( '%s_binary_path' % binary ):
       # The user has explicitly specified a path.
-      if os.path.isfile( user_options[ 'gocode_binary_path' ] ):
-        return user_options[ 'gocode_binary_path' ]
+      if os.path.isfile( user_options[ '%s_binary_path' % binary] ):
+        return user_options[ '%s_binary_path' % binary]
       else:
         return None
     # Try to use the bundled binary or one on the path.
-    if os.path.isfile( PATH_TO_GOCODE_BINARY ):
-      return PATH_TO_GOCODE_BINARY
-    return utils.PathToFirstExistingExecutable( [ 'gocode' ] )
+    if binary == 'gocode':
+      if os.path.isfile( PATH_TO_GOCODE_BINARY ):
+        return PATH_TO_GOCODE_BINARY
+    elif binary == 'godef':
+      if os.path.isfile( PATH_TO_GODEF_BINARY ):
+        return PATH_TO_GODEF_BINARY
+    return utils.PathToFirstExistingExecutable( [ binary ] )
 
 
   def OnFileReadyToParse( self, request_data ):
@@ -132,31 +156,66 @@ def Shutdown( self ):
 
   def _StartServer( self ):
     """ Start the GoCode server """
-    self._ExecuteGoCodeBinary()
+    self._ExecuteBinary( self._binary_gocode )
 
 
   def _StopServer( self ):
     """ Stop the GoCode server """
     _logger.info( 'Stopping GoCode server' )
-    self._ExecuteGoCodeBinary( 'close' )
+    self._ExecuteBinary( self._binary_gocode, 'close' )
 
 
-  def _ExecuteGoCodeBinary( self, *args, **kwargs ):
-    """ Execute the GoCode binary with given arguments. Use the contents
+  def _ExecuteBinary( self, binary, *args, **kwargs):
+    """ Execute the GoCode/GoDef binary with given arguments. Use the contents
     argument to send data to GoCode. Return the standard output. """
-    proc = self._popener(
-      [ self._binary ] + list(args), stdin = subprocess.PIPE,
+    popen_handle = self._popener(
+      [ binary ] + list(args), stdin = subprocess.PIPE,
       stdout = subprocess.PIPE, stderr = subprocess.PIPE )
     contents = kwargs[ 'contents' ] if 'contents' in kwargs else None
-    stdoutdata, stderrdata = proc.communicate( contents )
-    if proc.returncode:
-      _logger.error( COMPLETION_ERROR_MESSAGE + " code %i stderr: %s",
-                     proc.returncode, stderrdata)
-      raise RuntimeError( COMPLETION_ERROR_MESSAGE )
+    stdoutdata, stderrdata = popen_handle.communicate( contents )
+    if popen_handle.returncode:
+      binary_str = "Gocode"
+      if binary == self._binary_godef:
+        binary_str = "Godef"
+
+      _logger.error( SHELL_ERROR_MESSAGE.format( binary_str ) +
+                     " code %i stderr: %s",
+                     popen_handle.returncode, stderrdata)
+      raise RuntimeError( SHELL_ERROR_MESSAGE.format( binary_str )  )
 
     return stdoutdata
 
 
+  def _GoToDefinition( self, request_data ):
+    try:
+      filename = request_data[ 'filepath' ]
+      _logger.info( "godef GoTo request %s" % filename )
+      if not filename:
+        return
+      contents = utils.ToUtf8IfNeeded(
+          request_data[ 'file_data' ][ filename ][ 'contents' ] )
+      offset = _ComputeOffset( contents, request_data[ 'line_num' ],
+                               request_data[ 'column_num' ] )
+      stdout = self._ExecuteBinary( self._binary_godef,
+                                    "-i",
+                                    "-f=%s" % filename,
+                                    '-json',
+                                    "-o=%s" % offset,
+                                    contents = contents )
+      return self._ConstructGoToFromResponse( stdout )
+    except Exception:
+      raise RuntimeError( 'Can\'t jump to definition.' )
+
+
+  def _ConstructGoToFromResponse( self, response_str ):
+    parsed = json.loads( response_str )
+    if 'filename' in parsed and 'column' in parsed:
+      return responses.BuildGoToResponse( parsed[ 'filename' ],
+                                          int( parsed[ 'line' ] ),
+                                          int( parsed[ 'column' ] ) )
+    raise RuntimeError( 'Can\'t jump to definition.' )
+
+
 
 # Compute the byte offset in the file given the line and column.
 # TODO(ekfriis): If this is slow, consider moving this to C++ ycm_core,


Test patch:
diff --git a/ycmd/completers/go/tests/gocode_completer_test.py b/ycmd/completers/go/tests/gocode_completer_test.py
--- a/ycmd/completers/go/tests/gocode_completer_test.py
+++ b/ycmd/completers/go/tests/gocode_completer_test.py
@@ -65,15 +65,15 @@ def FindGoCodeBinary_test( self ):
     user_options = user_options_store.DefaultOptions()
 
     eq_( PATH_TO_GOCODE_BINARY,
-         self._completer.FindGoCodeBinary( user_options ) )
+         self._completer.FindBinary( "gocode", user_options ) )
 
     user_options[ 'gocode_binary_path' ] = DUMMY_BINARY
     eq_( DUMMY_BINARY,
-         self._completer.FindGoCodeBinary( user_options ) )
+         self._completer.FindBinary( "gocode", user_options ) )
 
     user_options[ 'gocode_binary_path' ] = DATA_DIR
     eq_( None,
-         self._completer.FindGoCodeBinary( user_options ) )
+         self._completer.FindBinary( "gocode", user_options ) )
 
 
   # Test line-col to offset in the file before any unicode occurrences.
diff --git a/ycmd/tests/go/subcommands_test.py b/ycmd/tests/go/subcommands_test.py
new file mode 100644
--- /dev/null
+++ b/ycmd/tests/go/subcommands_test.py
@@ -0,0 +1,75 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2016 ycmd contributors.
+#
+# This file is part of ycmd.
+#
+# ycmd is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# ycmd is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with ycmd.  If not, see <http://www.gnu.org/licenses/>.
+
+from go_handlers_test import Go_Handlers_test
+from nose.tools import eq_
+
+
+class Go_Subcommands_test( Go_Handlers_test ):
+
+
+  def _GoTo( self, params ):
+    filepath = self._PathToTestFile( 'goto.go' )
+    contents = open( filepath ).read()
+
+    command = params[ 'command' ]
+    goto_data = self._BuildRequest( completer_target = 'filetype_default',
+                                    command_arguments = [ command ],
+                                    line_num = 8,
+                                    column_num = 8,
+                                    contents = contents,
+                                    filetype = 'go',
+                                    filepath = filepath )
+
+    results = self._app.post_json( '/run_completer_command',
+                                   goto_data )
+
+    eq_( {
+      'line_num': 3, 'column_num': 6, 'filepath': filepath
+    }, results.json )
+
+    filepath = self._PathToTestFile( 'win.go' )
+    contents = open( filepath ).read()
+
+    command = params[ 'command' ]
+    goto_data = self._BuildRequest( completer_target = 'filetype_default',
+                                    command_arguments = [ command ],
+                                    line_num = 4,
+                                    column_num = 7,
+                                    contents = contents,
+                                    filetype = 'go',
+                                    filepath = filepath )
+
+    results = self._app.post_json( '/run_completer_command',
+                                   goto_data )
+
+    eq_( {
+      'line_num': 2, 'column_num': 6, 'filepath': filepath
+    }, results.json )
+
+
+  def GoTo_all_test( self ):
+    tests = [
+      { 'command': 'GoTo' },
+      { 'command': 'GoToDefinition' },
+      { 'command': 'GoToDeclaration' }
+    ]
+
+    for test in tests:
+      yield ( self._GoTo, test )
diff --git a/ycmd/tests/go/testdata/goto.go b/ycmd/tests/go/testdata/goto.go
new file mode 100644
--- /dev/null
+++ b/ycmd/tests/go/testdata/goto.go
@@ -0,0 +1,9 @@
+package main
+
+func dummy() {
+
+}
+
+func main() {
+    dummy() //GoTo
+}
\ No newline at end of file
diff --git a/ycmd/tests/go/testdata/win.go b/ycmd/tests/go/testdata/win.go
new file mode 100644
--- /dev/null
+++ b/ycmd/tests/go/testdata/win.go
@@ -0,0 +1,5 @@
+package main
+func foo() {}
+func main() {
+    foo()
+}