Initial, basic, heuristic, clojure analyzer

Hey guys,

I've been hacking on a clojure source analyzer.

After some research, turns out the frontend of the clojure compiler (well sort of, there may in the future be a move to this - for now the compiler is a sealed unit, but this emulates it exactly) is completely open and accessible. https://github.com/clojure/tools.analyzer.jvm Using this we can get a (very detailed) AST that we can use to build our data model. To do this though you have to, essentially, compile the input code. Obviously this requires all files for the project and all dependencies to be fed in atomically. It allows us to be completely semantic but we would need to load at the project-level and understand the build system etc. Promising that it is at least possible without requiring us to write a compiler. It looks like similar components are available for Java.

For now, I stuck with loading at the file level. This is all pretty basic. It should do a reasonable job of catching simple definitions and namespacing them. Here is a sample, at the REPL, on the analyze/clojure.clj file from this pull request:

    pathfinder.analyze> (def file (slurp "pathfinder/src/pathfinder/analyze/clojure.clj"))
    pathfinder.analyze> (analyze file {:type :clojure :path "/path/to/analyze/clojure.clj"})
    {:meta {:path "/path/to/analyze/clojure.clj", :type :clojure},
     :source
     "(ns pathfinder.analyze.clojure\n  (:require [pathfinder.analyze.analyzer :refer [Analyzer]]\n            [clojure.java.io :as io]\n            [clojure.tools.reader.reader-types :as readers]\n            [clojure.tools.reader :as reader]\n            [clojure.tools.analyzer.jvm :as jvm-analyzer]))\n\n;; TODO: arbitrary code can be executed during read (and obviously\n;; evaluation) of clojure code. Currently, these analyzers should only\n;; be used for trusted source. We should look in to sandboxing this\n;; process and probably killing it after a timeout.\n\n(defn- read-forms [source]\n  (let [pbr (readers/indexing-push-back-reader source)\n        eof (Object.)]\n    (take-while #(not (identical? % eof))\n                (repeatedly #(reader/read pbr false eof)))))\n\n(defn- extract-data-model [ast meta])\n\n;; TODO: using jvm-analyzer we can do a full parse and build a\n;; complete, perfectly accurate, ast. This is essentially the front half\n;; of the clojure compiler. It would imply loading code in at the\n;; granularity of a project. We would probably need to understand the\n;; build system too. For now I've hacked up a fairly basic parser based\n;; on heuristics, see CljHeuristicAnalyzer below.\n\n(deftype CljAnalyzer []\n  Analyzer\n  (analyze [this source meta]\n    (-> source\n        read-forms\n        (->> (map jvm-analyzer/analyze))\n        #_(extract-data-model meta))))\n\n(defn- extract-ns [state form]\n  (if (and (list? form) (= (first form) 'ns))\n    (assoc state :ns (second form))\n    state))\n\n(defn- extract-definition [state form]\n  ;; support only the clojure.core types for now\n  (let [types '{def :definition\n                definline :function\n                definterface :interface\n                defmacro :macro\n                defmethod :method\n                defmulti :multimethod\n                defn :function\n                defn- :function\n                defprotocol :protocol\n                defrecord :record\n                defstruct :struct\n                deftype :class}]\n    (if (and (list? form) (.startsWith (str (first form)) \"def\")) ;crude\n      (update-in state [:model :definitions]\n                 conj {:name (str (:ns state) (if (:ns state) \"/\" \"\") (second form))\n                       :type (get types (first form) :unknown)\n                       :pos (select-keys (meta form) [:column :line :end-column :end-line])})\n      state)))\n\n;;; TODO:\n(defn- extract-usages [state form] state)\n\n(defn- guess-data-model [forms]\n  (->> forms\n       (reduce (fn [state form]\n                 (-> state\n                     (extract-ns form)\n                     (extract-definition form)\n                     (extract-usages form)))\n               {:ns nil :model {}})\n       :model))\n\n(deftype CljHeuristicAnalyzer []\n  ;; guess without full parse and eval context\n  Analyzer\n  (analyze [this source meta]\n    (-> source\n        read-forms\n        guess-data-model\n        (merge {:source source\n                :meta meta}))))\n",
     :definitions
     ({:name "pathfinder.analyze.clojure/CljHeuristicAnalyzer",
       :type :class,
       :pos {:end-line 83, :end-column 32, :line 75, :column 1}}
      {:name "pathfinder.analyze.clojure/guess-data-model",
       :type :function,
       :pos {:end-line 73, :end-column 16, :line 65, :column 1}}
      {:name "pathfinder.analyze.clojure/extract-usages",
       :type :function,
       :pos {:end-line 63, :end-column 42, :line 63, :column 1}}
      {:name "pathfinder.analyze.clojure/extract-definition",
       :type :function,
       :pos {:end-line 60, :end-column 15, :line 41, :column 1}}
      {:name "pathfinder.analyze.clojure/extract-ns",
       :type :function,
       :pos {:end-line 39, :end-column 12, :line 36, :column 1}}
      {:name "pathfinder.analyze.clojure/CljAnalyzer",
       :type :class,
       :pos {:end-line 34, :end-column 39, :line 28, :column 1}}
      {:name "pathfinder.analyze.clojure/extract-data-model",
       :type :function,
       :pos {:end-line 19, :end-column 38, :line 19, :column 1}}
      {:name "pathfinder.analyze.clojure/read-forms",
       :type :function,
       :pos {:end-line 17, :end-column 61, :line 13, :column 1}})}

What do you think of the data model? I'll look at trying to extract all usages of functions so we can find references. Shouldn't be too hard. Again, it'll be heuristic-based. Interested to gather some feedback on this.

Cheers,

-G